[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 474691975 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003332] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010134] LSM: Security Framework initializing [ 0.012031] Yama: becoming mindful. [ 0.013035] SELinux: Initializing. [ 0.014074] *** VALIDATE selinux *** [ 0.022766] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027159] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028154] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029112] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030106] *** VALIDATE tmpfs *** [ 0.031434] *** VALIDATE proc *** [ 0.032206] *** VALIDATE cgroup *** [ 0.033011] *** VALIDATE cgroup2 *** [ 0.035083] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036152] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038030] Spectre V2 : User space: Vulnerable [ 0.039012] Speculative Store Bypass: Vulnerable [ 0.042478] debug: unmapping init [mem 0xffffffffade59000-0xffffffffade60fff] [ 0.044971] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045669] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046021] ... version: 2 [ 0.047013] ... bit width: 48 [ 0.048019] ... generic registers: 4 [ 0.049011] ... value mask: 0000ffffffffffff [ 0.050013] ... max period: 00007fffffffffff [ 0.051013] ... fixed-purpose events: 3 [ 0.052013] ... event mask: 000000070000000f [ 0.053283] rcu: Hierarchical SRCU implementation. [ 0.055385] smp: Bringing up secondary CPUs ... [ 0.056553] x86: Booting SMP configuration: [ 0.057022] .... node #0, CPUs: #1 #2 #3 [ 0.060016] smp: Brought up 1 node, 4 CPUs [ 0.062019] smpboot: Max logical packages: 1 [ 0.063021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.138378] node 0 deferred pages initialised in 72ms [ 0.144158] devtmpfs: initialized [ 0.145228] x86/mm: Memory block size: 128MB [ 0.147823] gcov: version magic: 0x41383552 [ 0.149160] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.152084] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.154290] pinctrl core: initialized pinctrl subsystem [ 0.156136] [ 0.156552] ************************************************************* [ 0.158011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.161012] ** ** [ 0.162010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.165012] ** ** [ 0.167011] ** This means that this kernel is built to expose internal ** [ 0.169012] ** IOMMU data structures, which may compromise security on ** [ 0.171009] ** your system. ** [ 0.173010] ** ** [ 0.175011] ** If you see this message and you are not debugging the ** [ 0.177013] ** kernel, report this immediately to your vendor! ** [ 0.179014] ** ** [ 0.182013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.184012] ************************************************************* [ 0.186732] NET: Registered protocol family 16 [ 0.188414] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.190056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.193060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.196420] cpuidle: using governor menu [ 0.197826] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.200430] PCI: Using configuration type 1 for base access [ 0.202114] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.212118] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.215055] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.219120] cryptd: max_cpu_qlen set to 1000 [ 0.222225] ACPI: Added _OSI(Module Device) [ 0.223014] ACPI: Added _OSI(Processor Device) [ 0.225012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.226013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.230891] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.237540] ACPI: Interpreter enabled [ 0.240059] ACPI: PM: (supports S0 S3 S4 S5) [ 0.241009] ACPI: Using IOAPIC for interrupt routing [ 0.243088] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.247382] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.256000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.258041] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.260018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.263098] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.268757] acpiphp: Slot [2] registered [ 0.270104] acpiphp: Slot [5] registered [ 0.272122] acpiphp: Slot [6] registered [ 0.273121] acpiphp: Slot [7] registered [ 0.275111] acpiphp: Slot [8] registered [ 0.276124] acpiphp: Slot [9] registered [ 0.277099] acpiphp: Slot [10] registered [ 0.279098] acpiphp: Slot [3] registered [ 0.280082] acpiphp: Slot [4] registered [ 0.281102] acpiphp: Slot [11] registered [ 0.283092] acpiphp: Slot [12] registered [ 0.284089] acpiphp: Slot [13] registered [ 0.285120] acpiphp: Slot [14] registered [ 0.287118] acpiphp: Slot [15] registered [ 0.288094] acpiphp: Slot [16] registered [ 0.290124] acpiphp: Slot [17] registered [ 0.291095] acpiphp: Slot [18] registered [ 0.293133] acpiphp: Slot [19] registered [ 0.294111] acpiphp: Slot [20] registered [ 0.295106] acpiphp: Slot [21] registered [ 0.297097] acpiphp: Slot [22] registered [ 0.298088] acpiphp: Slot [23] registered [ 0.299071] acpiphp: Slot [24] registered [ 0.299947] acpiphp: Slot [25] registered [ 0.301115] acpiphp: Slot [26] registered [ 0.302077] acpiphp: Slot [27] registered [ 0.304074] acpiphp: Slot [28] registered [ 0.305133] acpiphp: Slot [29] registered [ 0.307101] acpiphp: Slot [30] registered [ 0.308100] acpiphp: Slot [31] registered [ 0.310060] PCI host bridge to bus 0000:00 [ 0.311016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.313024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.315022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.317021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.320021] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.322020] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.324192] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.326957] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.330240] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.339752] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.344056] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.346016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.348018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.350017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.353369] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.356774] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.357041] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.359918] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.365014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.376017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.381015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.387823] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.393015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.399013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.414016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.425540] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.434013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.439012] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.458013] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.466590] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.473014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.478012] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.499015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.507839] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.513014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.520014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.535016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.544828] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.551014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.559014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.578014] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.589367] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.594011] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.598012] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.613018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.622108] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.624282] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.625323] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.628370] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.630186] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.635044] iommu: Default domain type: Passthrough [ 0.636367] SCSI subsystem initialized [ 0.638107] ACPI: bus type USB registered [ 0.639109] usbcore: registered new interface driver usbfs [ 0.640062] usbcore: registered new interface driver hub [ 0.642084] usbcore: registered new device driver usb [ 0.643152] pps_core: LinuxPPS API ver. 1 registered [ 0.645011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.647056] PTP clock support registered [ 0.650073] EDAC MC: Ver: 3.0.0 [ 0.652135] PCI: Using ACPI for IRQ routing [ 0.653755] NetLabel: Initializing [ 0.654007] NetLabel: domain hash size = 128 [ 0.655009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.657070] NetLabel: unlabeled traffic allowed by default [ 0.658125] vgaarb: loaded [ 0.661257] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.662009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.667597] clocksource: Switched to clocksource kvm-clock [ 0.769842] VFS: Disk quotas dquot_6.6.0 [ 0.771303] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.773464] *** VALIDATE ramfs *** [ 0.774385] *** VALIDATE hugetlbfs *** [ 0.775478] pnp: PnP ACPI init [ 0.777319] pnp: PnP ACPI: found 6 devices [ 0.793541] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.796832] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.798890] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.800322] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.802558] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.805117] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.808104] NET: Registered protocol family 2 [ 0.810651] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.815596] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.817677] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.822284] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.825507] TCP: Hash tables configured (established 65536 bind 65536) [ 0.828459] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.831510] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.834093] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.836597] NET: Registered protocol family 1 [ 0.838353] RPC: Registered named UNIX socket transport module. [ 0.840422] RPC: Registered udp transport module. [ 0.842142] RPC: Registered tcp transport module. [ 0.843901] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.846383] NET: Registered protocol family 44 [ 0.847701] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.849770] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.852030] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.854170] PCI: CLS 0 bytes, default 64 [ 0.855729] Unpacking initramfs... [ 2.252433] debug: unmapping init [mem 0xffff901fbcc54000-0xffff901fbffbffff] [ 2.256225] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.258565] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.260797] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.740959] Initialise system trusted keyrings [ 2.742614] Key type blacklist registered [ 2.744471] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.756787] zbud: loaded [ 2.777540] *** VALIDATE nfs *** [ 2.778901] *** VALIDATE nfs4 *** [ 2.780597] pstore: using deflate compression [ 2.784440] Platform Keyring initialized [ 2.881908] NET: Registered protocol family 38 [ 2.883651] Key type asymmetric registered [ 2.885213] Asymmetric key parser 'x509' registered [ 2.887200] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.891249] io scheduler mq-deadline registered [ 2.893050] io scheduler kyber registered [ 2.894306] io scheduler bfq registered [ 2.896290] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.898925] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.901938] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.904289] ACPI: Power Button [PWRF] [ 2.910678] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.917706] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.934719] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.943757] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.965642] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.995768] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.024595] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.029247] Non-volatile memory driver v1.3 [ 3.030818] Linux agpgart interface v0.103 [ 3.062598] virtio_blk virtio1: [vda] 146152 512-byte logical blocks (74.8 MB/71.4 MiB) [ 3.066203] vda: detected capacity change from 0 to 74829824 [ 3.082053] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.085195] vdb: detected capacity change from 0 to 1073741824 [ 3.101157] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.104396] vdc: detected capacity change from 0 to 2621440000 [ 3.119996] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.123123] vdd: detected capacity change from 0 to 2621440000 [ 3.137855] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.141365] vde: detected capacity change from 0 to 4294967296 [ 3.159460] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.163246] vdf: detected capacity change from 0 to 4294967296 [ 3.171120] libphy: Fixed MDIO Bus: probed [ 3.175975] usbcore: registered new interface driver usbserial_generic [ 3.178569] usbserial: USB Serial support registered for generic [ 3.180911] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.185800] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.187697] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.190439] mousedev: PS/2 mouse device common for all mice [ 3.193764] rtc_cmos 00:05: RTC can wake from S4 [ 3.198935] rtc_cmos 00:05: registered as rtc0 [ 3.198992] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.200452] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.206864] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.207146] intel_pstate: CPU model not supported [ 3.212474] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.217213] hid: raw HID events driver (C) Jiri Kosina [ 3.219243] usbcore: registered new interface driver usbhid [ 3.221260] usbhid: USB HID core driver [ 3.222895] drop_monitor: Initializing network drop monitor service [ 3.225475] Initializing XFRM netlink socket [ 3.227279] NET: Registered protocol family 10 [ 3.230430] Segment Routing with IPv6 [ 3.231975] NET: Registered protocol family 17 [ 3.234244] mpls_gso: MPLS GSO support [ 3.240100] RAS: Correctable Errors collector initialized. [ 3.242397] AVX version of gcm_enc/dec engaged. [ 3.244186] AES CTR mode by8 optimization enabled [ 3.324081] sched_clock: Marking stable (3324036839, 0)->(4234078003, -910041164) [ 3.327342] registered taskstats version 1 [ 3.329356] Loading compiled-in X.509 certificates [ 3.331319] zswap: loaded using pool lzo/zbud [ 3.357716] Key type big_key registered [ 3.371135] Key type encrypted registered [ 3.372793] ima: No TPM chip found, activating TPM-bypass! [ 3.375076] ima: Allocated hash algorithm: sha1 [ 3.376948] ima: No architecture policies found [ 3.378441] evm: Initialising EVM extended attributes: [ 3.380337] evm: security.selinux [ 3.381426] evm: security.ima [ 3.382463] evm: security.capability [ 3.383820] evm: HMAC attrs: 0x1 [ 3.386190] rtc_cmos 00:05: setting system clock to 2026-08-24 12:30:16 UTC (1787574616) [ 3.392539] debug: unmapping init [mem 0xffffffffaee03000-0xffffffffaeffffff] [ 3.395535] debug: unmapping init [mem 0xffffffffadb82000-0xffffffffade58fff] [ 3.405329] Write protecting the kernel read-only data: 28672k [ 3.408763] debug: unmapping init [mem 0xffffffffac203000-0xffffffffac3fffff] [ 3.411984] debug: unmapping init [mem 0xffffffffacb14000-0xffffffffacbfffff] [ 3.448673] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.457692] systemd[1]: Detected virtualization kvm. [ 3.460257] systemd[1]: Detected architecture x86-64. [ 3.462345] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.489199] systemd[1]: No hostname configured. [ 3.491098] systemd[1]: Set hostname to . [ 3.493733] random: systemd: uninitialized urandom read (16 bytes read) [ 3.496516] systemd[1]: Initializing machine ID from random generator. [ 3.547544] random: ln: uninitialized urandom read (6 bytes read) [ 3.646646] random: systemd: uninitialized urandom read (16 bytes read) [ 3.649473] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.654301] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.664399] systemd[1]: Starting Apply Kernel Variables... Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.339873] device-mapper: uevent: version 1.0.3 [ 4.342270] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices...[ 4.744822] random: fast init done Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.154852] virtio_net virtio0 ens2: renamed from eth0 [ 5.227129] scsi host0: ata_piix [ 5.232270] scsi host1: ata_piix [ 5.233726] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.235673] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.039859] dracut-initqueue[587]: RTNETLINK answers: File exists [ 9.888271] random: crng init done [ 9.889615] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.442878] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. Stopping udev Kernel Device Manager... [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.598323] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.862287] SELinux: Disabled at runtime. [ 11.924263] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.932787] systemd[1]: Detected virtualization kvm. [ 11.934763] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.488898] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.492589] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.497633] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.502433] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.506602] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.514521] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.522158] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ 12.556742] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Mounting Huge Pages File System... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.022521] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.267091] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.342867] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.473314] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.500713] EDAC sbridge: Ver: 1.1.2 [ 15.381834] Key type dns_resolver registered [ 15.693955] NFS: Registering the id_resolver key type [ 15.695941] Key type id_resolver registered [ 15.697503] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg612-server login: [ 41.791990] libcfs: loading out-of-tree module taints kernel. [ 41.813293] Key type ._llcrypt registered [ 41.814824] Key type .llcrypt registered [ 41.870524] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [ 50.143391] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 50.911591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 50.919702] alg: No test for adler32 (adler32-zlib) [ 51.969787] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [ 53.110081] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 54.983599] Key type lgssc registered [ 56.821272] Lustre: Echo OBD driver; http://www.lustre.org/ [ 64.930006] hrtimer: interrupt took 8997104 ns [ 74.189303] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 117.441255] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 131.938796] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 131.992713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 133.636413] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 133.699937] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 133.894111] Lustre: lustre-MDT0000: new disk, initializing [ 134.039811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 134.065217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 139.869059] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 156.100917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 156.246559] Lustre: 6493:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 156.367104] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 156.372465] Lustre: Skipped 1 previous similar message [ 156.512565] Lustre: lustre-MDT0001: new disk, initializing [ 156.599744] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 156.625190] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 156.654638] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 161.412606] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 166.838742] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 177.704346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 178.018966] Lustre: lustre-OST0000: new disk, initializing [ 178.024296] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 178.030936] Lustre: 8436:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 178.084571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 183.332112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 183.346319] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 183.380634] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 183.550930] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 197.793794] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 197.959701] Lustre: lustre-OST0001: new disk, initializing [ 197.966664] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 197.971771] Lustre: 9505:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 198.060110] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 203.816664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 203.828955] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 203.886958] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 204.943099] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 215.724303] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 222.757717] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 228.812923] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing check_logdir /tmp/testlogs/ [ 233.276514] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing yml_node [ 237.687796] Lustre: DEBUG MARKER: Client: 2.17.57.80 [ 240.160824] Lustre: DEBUG MARKER: MDS: 2.17.57.80 [ 242.634825] Lustre: DEBUG MARKER: OSS: 2.17.57.80 [ 244.160920] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 24 08:34:16 EDT 2026 [ 260.346577] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 261.723861] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 279.520606] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 279.527127] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 279.545908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 280.548784] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 280.551039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 280.581409] Lustre: Skipped 1 previous similar message [ 280.596578] Lustre: Skipped 3 previous similar messages [ 282.962581] Lustre: server umount lustre-MDT0000 complete [ 285.672808] LustreError: 10698:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 285.687230] LustreError: 10698:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 286.311322] LustreError: 9504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787574899 with bad export cookie 17295004872294738001 [ 286.312722] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 286.321874] LustreError: 9504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 290.785064] LustreError: 6500:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 290.787195] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 290.795589] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 290.802680] LustreError: 6500:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 290.819642] Lustre: Skipped 2 previous similar messages [ 291.808818] LustreError: 7740:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 291.826087] LustreError: 7740:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 292.625248] Lustre: server umount lustre-MDT0001 complete [ 296.637549] Lustre: server umount lustre-OST0000 complete [ 300.162595] Lustre: server umount lustre-OST0001 complete [ 307.733538] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [ 316.092886] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 348.505259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 348.710365] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 348.752502] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 348.895442] Lustre: lustre-MDT0000: new disk, initializing [ 349.074467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 349.094050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 353.501197] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 363.172800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 363.331229] Lustre: 15056:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 363.422469] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 363.426147] Lustre: Skipped 1 previous similar message [ 363.484416] Lustre: lustre-MDT0001: new disk, initializing [ 363.585381] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 363.633725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 363.649981] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 367.762526] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 377.190938] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 383.289554] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 390.773319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 391.182456] Lustre: lustre-OST0000: new disk, initializing [ 391.187426] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 391.199307] Lustre: 17026:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 391.287891] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 392.709473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 392.724684] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 392.896988] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 397.931781] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 408.160834] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 413.549272] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 413.746580] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 419.144396] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 419.323795] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 423.394868] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 423.399069] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 423.410178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 423.410185] Lustre: Skipped 3 previous similar messages [ 423.432956] Lustre: Skipped 1 previous similar message [ 427.183795] Lustre: server umount lustre-OST0000 complete [ 433.645859] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 433.650093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 433.677697] Lustre: Skipped 1 previous similar message [ 433.696929] Lustre: Skipped 2 previous similar messages [ 437.456168] Lustre: server umount lustre-MDT0000 complete [ 438.756274] LustreError: 15069:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 438.783687] LustreError: 15069:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 441.414878] LustreError: 15050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787575054 with bad export cookie 17295004872294739534 [ 441.422654] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 441.434167] LustreError: 15050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 441.979761] Lustre: server umount lustre-MDT0001 complete [ 445.999792] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 08:37:37 (1787575057) [ 448.383255] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 449.993953] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 08:37:41 (1787575061) [ 456.214895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 456.738770] LustreError: 19435:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 456.848869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 461.531941] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 466.473205] LustreError: 19435:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 466.492133] LustreError: 19435:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 470.295096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 475.564493] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 483.108654] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 488.669268] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 495.047306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 501.143256] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 509.434989] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 522.210431] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 522.223917] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 522.253375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 522.262496] Lustre: Skipped 1 previous similar message [ 525.189634] Lustre: server umount lustre-OST0000 complete [ 532.449089] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 532.464799] LustreError: Skipped 1 previous similar message [ 536.099485] Lustre: server umount lustre-MDT0000 complete [ 537.569529] LustreError: 19436:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 537.587458] LustreError: 19436:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 539.545938] LustreError: 22217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787575152 with bad export cookie 17295004872294740409 [ 539.549330] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 539.553816] LustreError: 22217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 539.926223] Lustre: server umount lustre-MDT0001 complete [ 550.757969] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 553.494506] Key type lgssc unregistered [ 553.743787] LNet: 22971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 553.751244] LNetError: 22971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 553.783091] LNet: Removed LNI 192.168.206.112@tcp [ 554.682157] Key type .llcrypt unregistered [ 554.683979] Key type ._llcrypt unregistered [ 576.380280] Key type ._llcrypt registered [ 576.383395] Key type .llcrypt registered [ 576.549800] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 578.031312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 578.103352] alg: No test for adler32 (adler32-zlib) [ 579.353655] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [ 579.588216] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 581.384610] Key type lgssc registered [ 582.649443] Lustre: Echo OBD driver; http://www.lustre.org/ [ 590.762557] loop: module loaded [ 590.851172] loop0: detected capacity change from 0 to 204800000 [ 591.160107] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 591.173904] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 591.551789] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 597.398214] loop0: detected capacity change from 0 to 204800000 [ 597.577744] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 597.589666] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 597.597460] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 602.173223] loop0: detected capacity change from 0 to 204800000 [ 607.222113] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 607.240278] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 608.595088] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 608.688173] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 608.803948] Lustre: lustre-MDT0000: new disk, initializing [ 608.910231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 608.946320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 613.625729] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 621.273736] loop1: detected capacity change from 0 to 204800000 [ 626.219548] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 626.441256] Lustre: lustre-OST07c6: new disk, initializing [ 626.444682] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 626.448503] Lustre: Skipped 1 previous similar message [ 626.453223] Lustre: 26157:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 626.523596] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 632.138219] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 636.029401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 636.039618] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 636.075756] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 641.783553] Lustre: 25144:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 641.794589] Lustre: 25144:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 641.804743] Lustre: 25144:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 641.812443] Lustre: 25144:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 641.825948] Lustre: 25144:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 641.834137] Lustre: 25144:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 661.097135] Lustre: 25146:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 661.113288] Lustre: 25146:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 661.124359] Lustre: 25146:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 661.139229] Lustre: 25146:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 661.145564] Lustre: 25146:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 661.152866] Lustre: 25146:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 665.737925] Lustre: 25146:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 665.747813] Lustre: 25146:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 665.756552] Lustre: 25146:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 665.768841] Lustre: 25146:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 665.776481] Lustre: 25146:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 665.792723] Lustre: 25146:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 665.801634] Lustre: 25146:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 665.809703] Lustre: 25146:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 665.815821] Lustre: 25146:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 665.821155] Lustre: 25146:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 665.825759] Lustre: 25146:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 665.831638] Lustre: 25146:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 668.985218] Lustre: 25144:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 668.996305] Lustre: 25144:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 668.999664] Lustre: 25144:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 669.004869] Lustre: 25144:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 669.011208] Lustre: 25144:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 669.016488] Lustre: 25144:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 669.023133] Lustre: 25144:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 669.029565] Lustre: 25144:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 669.038204] Lustre: 25144:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 669.044413] Lustre: 25144:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 669.053624] Lustre: 25144:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 669.066174] Lustre: 25144:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 673.367816] Lustre: 25145:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 673.372455] Lustre: 25145:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 673.379184] Lustre: 25145:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 673.383860] Lustre: 25145:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 673.390607] Lustre: 25145:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 673.397361] Lustre: 25145:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 673.403630] Lustre: 25145:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 673.420023] Lustre: 25145:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 673.426631] Lustre: 25145:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 673.436547] Lustre: 25145:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 673.446349] Lustre: 25145:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 673.459479] Lustre: 25145:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 685.573188] Lustre: 26673:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 685.588590] Lustre: 26673:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 685.598651] Lustre: 26673:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 685.605284] Lustre: 26673:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 685.611258] Lustre: 26673:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 685.618054] Lustre: 26673:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 685.625195] Lustre: 26673:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 685.633601] Lustre: 26673:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 685.637474] Lustre: 26673:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 685.641338] Lustre: 26673:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 685.645298] Lustre: 26673:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 685.649176] Lustre: 26673:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 733.160635] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 733.169983] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 733.180705] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 738.275351] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 738.817691] Lustre: server umount lustre-OST07c6 complete [ 748.097766] Lustre: server umount lustre-MDT0000 complete [ 765.314459] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 768.241413] Key type lgssc unregistered [ 768.565974] LNet: 28720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 768.578091] LNetError: 28720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 768.611986] LNet: Removed LNI 192.168.206.112@tcp [ 769.428121] Key type .llcrypt unregistered [ 769.434447] Key type ._llcrypt unregistered [ 787.714133] Key type ._llcrypt registered [ 787.717351] Key type .llcrypt registered [ 787.817502] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 788.545558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 788.647965] alg: No test for adler32 (adler32-zlib) [ 789.665837] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [ 789.961256] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 791.647186] Key type lgssc registered [ 792.824282] Lustre: Echo OBD driver; http://www.lustre.org/ [ 800.063759] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 800.090624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 801.583767] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 805.943401] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 810.979697] LustreError: 29992:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 811.001842] LustreError: 29992:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 813.812908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 814.272795] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 819.027717] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 827.061677] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 832.759521] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 839.865822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 840.485943] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 846.655059] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 847.299186] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 854.096809] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 872.419318] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 872.425812] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 872.458707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 874.464565] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 874.465981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 874.479329] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 874.502198] Lustre: Skipped 1 previous similar message [ 877.155917] Lustre: server umount lustre-OST0000 complete [ 884.719047] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 884.754358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 884.764546] Lustre: Skipped 1 previous similar message [ 887.118163] Lustre: server umount lustre-MDT0000 complete [ 889.824497] LustreError: 29987:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 889.846455] LustreError: 29987:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 890.801176] LustreError: 29974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787575503 with bad export cookie 16372993157319381514 [ 890.804227] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 890.814652] LustreError: 29974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 891.244935] Lustre: server umount lustre-MDT0001 complete [ 900.110643] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 902.962646] Key type lgssc unregistered [ 903.260763] LNet: 33732:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 903.264795] LNetError: 33732:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 903.285566] LNet: Removed LNI 192.168.206.112@tcp [ 904.125679] Key type .llcrypt unregistered [ 904.130445] Key type ._llcrypt unregistered [ 925.011604] Key type ._llcrypt registered [ 925.014299] Key type .llcrypt registered [ 925.148522] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 925.986887] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 926.032149] alg: No test for adler32 (adler32-zlib) [ 927.486948] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [ 927.954877] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 929.706994] Key type lgssc registered [ 931.380691] Lustre: Echo OBD driver; http://www.lustre.org/ [ 940.898985] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 940.916560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 942.575703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 947.711255] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 952.802980] LustreError: 35000:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 952.825120] LustreError: 35000:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 957.291546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 957.843797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 963.007859] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 970.575249] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 975.634496] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 982.662677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 983.189873] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 987.346994] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 992.178592] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 1000.212660] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1013.224352] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1013.230645] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1013.244216] LustreError: Skipped 1 previous similar message [ 1013.251954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1013.266893] Lustre: Skipped 1 previous similar message [ 1016.410559] Lustre: server umount lustre-OST0000 complete [ 1023.464493] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1023.474563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1023.483212] Lustre: Skipped 2 previous similar messages [ 1023.507554] Lustre: Skipped 1 previous similar message [ 1026.904721] Lustre: server umount lustre-MDT0000 complete [ 1028.577924] LustreError: 35000:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1028.601840] LustreError: 35000:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1030.963564] LustreError: 37497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787575644 with bad export cookie 15748997543747131505 [ 1030.967268] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1030.977617] LustreError: 37497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1031.451325] Lustre: server umount lustre-MDT0001 complete [ 1044.539945] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 1048.256968] Key type lgssc unregistered [ 1048.650857] LNet: 38539:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1048.665324] LNetError: 38539:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1048.682995] LNet: Removed LNI 192.168.206.112@tcp [ 1049.539248] Key type .llcrypt unregistered [ 1049.541876] Key type ._llcrypt unregistered [ 1072.889286] Key type ._llcrypt registered [ 1072.891320] Key type .llcrypt registered [ 1072.999482] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 1074.201293] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1074.440700] alg: No test for adler32 (adler32-zlib) [ 1075.476779] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [ 1075.820224] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 1077.591190] Key type lgssc registered [ 1079.513239] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1087.282256] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1087.310948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1088.878404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1093.667322] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 1098.210280] LustreError: 39972:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1098.236979] LustreError: 39972:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1102.759990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1103.161039] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1107.617715] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 1115.141486] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1120.720866] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1126.987087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1127.452268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1133.623367] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1134.184307] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 1141.116703] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1147.870683] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.12@tcp (stopping) [ 1148.899692] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1148.900867] Lustre: server umount lustre-MDT0000 complete [ 1148.921173] Lustre: Skipped 2 previous similar messages [ 1148.921683] LustreError: 41212:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1152.708689] LustreError: 39952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787575765 with bad export cookie 9971443229171272617 [ 1152.714192] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1152.720754] LustreError: 39952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1152.958578] LustreError: 41212:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1152.985233] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.12@tcp (stopping) [ 1152.989757] LustreError: 41212:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1154.023135] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1154.032996] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1157.025148] LustreError: 39965:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1157.027227] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1157.044393] LustreError: 39965:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1162.147603] LustreError: 39965:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1162.148248] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1162.165479] LustreError: 39965:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1162.186370] Lustre: Skipped 1 previous similar message [ 1167.327240] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1167.617268] Lustre: server umount lustre-MDT0001 complete [ 1182.733066] Lustre: server umount lustre-OST0000 complete [ 1199.380628] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 1202.618588] Key type lgssc unregistered [ 1202.957704] LNet: 43833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1202.976350] LNetError: 43833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1203.002740] LNet: Removed LNI 192.168.206.112@tcp [ 1203.935251] Key type .llcrypt unregistered [ 1203.937484] Key type ._llcrypt unregistered [ 1224.828261] Key type ._llcrypt registered [ 1224.829694] Key type .llcrypt registered [ 1225.018994] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 1226.851658] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1227.197292] alg: No test for adler32 (adler32-zlib) [ 1228.588204] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [ 1229.111966] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 1230.863190] Key type lgssc registered [ 1232.881041] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1242.774556] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1242.793153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1244.385399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1248.949498] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 1254.885957] LustreError: 45233:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1254.903918] LustreError: 45233:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1257.239491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1257.812713] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1263.022420] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 1271.319031] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1277.763128] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1286.936599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1287.217502] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1289.281135] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1293.313864] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 1301.159433] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1308.119241] Lustre: lustre-OST0000: Not available for connect from 192.168.206.12@tcp (stopping) [ 1309.674057] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1309.679962] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1309.703306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1312.740128] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1312.746959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1312.761968] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1312.785128] Lustre: Skipped 1 previous similar message [ 1317.857039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1317.872521] Lustre: Skipped 2 previous similar messages [ 1321.951335] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1322.179784] Lustre: server umount lustre-OST0000 complete [ 1343.137404] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1343.154092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1343.176016] Lustre: Skipped 2 previous similar messages [ 1346.313859] Lustre: server umount lustre-MDT0000 complete [ 1348.579906] LustreError: 45232:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1348.599202] LustreError: 45232:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1349.810352] LustreError: 45211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787575962 with bad export cookie 17552972124688644589 [ 1349.818281] LustreError: 45211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1349.824243] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1356.240504] Lustre: server umount lustre-MDT0001 complete [ 1367.151610] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 1369.972325] Key type lgssc unregistered [ 1370.264636] LNet: 48945:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1370.273591] LNetError: 48945:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1370.291978] LNet: Removed LNI 192.168.206.112@tcp [ 1371.160880] Key type .llcrypt unregistered [ 1371.163110] Key type ._llcrypt unregistered [ 1394.820469] Key type ._llcrypt registered [ 1394.823385] Key type .llcrypt registered [ 1394.995514] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 1395.849318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1396.021827] alg: No test for adler32 (adler32-zlib) [ 1397.255605] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [ 1397.503482] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 1399.223175] Key type lgssc registered [ 1400.586690] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1408.380983] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1408.413319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1409.857269] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1414.041125] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 1419.240917] LustreError: 50363:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1419.255582] LustreError: 50363:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1422.962201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1423.319652] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1427.599871] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 1436.101567] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1441.799454] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1448.454401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1448.777650] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1453.954319] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1455.207830] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 1462.315329] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1468.451257] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1472.056112] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1484.373625] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-24 8h54m55s [ 1486.243135] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1489.894503] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1489.912911] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1489.918471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1489.933910] Lustre: Skipped 2 previous similar messages [ 1490.977844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1490.988166] Lustre: Skipped 3 previous similar messages [ 1494.946725] Lustre: server umount lustre-MDT0000 complete [ 1496.528581] LustreError: 52852:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1498.655851] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1502.656286] LustreError: 50358:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1502.677094] LustreError: 50358:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1505.100767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1505.301400] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1505.573793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1505.641236] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1506.622959] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1510.233410] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 1513.548856] Lustre: DEBUG MARKER: Wait for df (36832) ... [ 1514.916930] Lustre: DEBUG MARKER: done [ 1521.121590] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1521.130281] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1521.156877] Lustre: Skipped 2 previous similar messages [ 1521.165972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1521.174321] Lustre: Skipped 1 previous similar message [ 1525.170888] Lustre: server umount lustre-OST0000 complete [ 1532.384414] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1532.394884] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1532.396588] LustreError: Skipped 1 previous similar message [ 1532.402836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1532.411369] Lustre: Skipped 2 previous similar messages [ 1532.435666] Lustre: Skipped 4 previous similar messages [ 1535.659587] Lustre: server umount lustre-MDT0000 complete [ 1539.553768] LustreError: 50358:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1539.570508] LustreError: 50358:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1539.760144] LustreError: 50343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787576152 with bad export cookie 556077192041701880 [ 1539.767599] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1539.773726] LustreError: 50343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1540.128765] Lustre: server umount lustre-MDT0001 complete [ 1552.193626] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 1555.118312] Key type lgssc unregistered [ 1555.435610] LNet: 55289:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1555.443150] LNetError: 55289:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1556.518425] LNet: Removed LNI 192.168.206.112@tcp [ 1557.421113] Key type .llcrypt unregistered [ 1557.423579] Key type ._llcrypt unregistered [ 1601.862662] Key type ._llcrypt registered [ 1601.866843] Key type .llcrypt registered [ 1602.008228] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 1603.515614] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1603.612354] alg: No test for adler32 (adler32-zlib) [ 1604.683214] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [ 1604.980223] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 1606.683371] Key type lgssc registered [ 1608.125904] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1616.882435] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1616.904253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1618.186726] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1618.221476] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1618.413951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1623.095756] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 1631.995927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1632.150404] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1632.185591] Lustre: 57499:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1632.508960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1636.582571] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 1645.095700] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1651.453608] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1659.742740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1660.060166] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1660.343140] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1666.679690] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1667.523614] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 1676.500642] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1684.948945] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-24 8h58m16s [ 1686.668435] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1695.205497] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1697.848360] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1697.927439] LustreError: 57507:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1718.266839] Lustre: lustre-MDT0000: Client 5b447d40-dfd3-4eb1-be08-8cb670843daf (at 192.168.206.12@tcp) reconnecting [ 1737.935124] LustreError: 57507:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1740.470914] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1743.173119] Lustre: DEBUG MARKER: done [ 1753.568972] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1753.585714] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1753.608346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1754.089409] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1754.103251] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1754.121508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1754.140055] Lustre: Skipped 1 previous similar message [ 1759.202118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1759.216549] Lustre: Skipped 1 previous similar message [ 1764.320718] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1764.326970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1764.337295] Lustre: Skipped 1 previous similar message [ 1764.481712] Lustre: server umount lustre-OST0000 complete [ 1769.440932] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1769.452976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1769.458899] Lustre: Skipped 1 previous similar message [ 1781.728758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1781.746868] Lustre: Skipped 5 previous similar messages [ 1782.239135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1782.454596] Lustre: server umount lustre-MDT0000 complete [ 1786.136526] LustreError: 57493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787576399 with bad export cookie 12142945456255240210 [ 1786.148153] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1786.155202] LustreError: 57493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1786.559490] Lustre: server umount lustre-MDT0001 complete [ 1797.646850] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 1800.569141] Key type lgssc unregistered [ 1800.901968] LNet: 61835:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1800.909050] LNetError: 61835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1800.942075] LNet: Removed LNI 192.168.206.112@tcp [ 1801.769941] Key type .llcrypt unregistered [ 1801.772410] Key type ._llcrypt unregistered [ 1845.138160] Key type ._llcrypt registered [ 1845.142349] Key type .llcrypt registered [ 1845.346380] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 1846.730845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1846.780854] alg: No test for adler32 (adler32-zlib) [ 1847.831519] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [ 1848.105524] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 1849.896751] Key type lgssc registered [ 1851.052136] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1854.274866] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1854.301327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1855.600899] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1855.640335] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1855.850678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1856.845057] Lustre: Failing over lustre-MDT0000 [ 1857.136489] Lustre: server umount lustre-MDT0000 complete [ 1864.123896] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 09:01:16 (1787576476) [ 1871.658910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1871.978851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1876.667874] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 1886.216125] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1886.310936] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1886.372318] Lustre: 64646:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1886.748504] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1892.452147] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 1903.416123] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1910.868601] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1918.255844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1918.542042] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1918.932785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1926.712025] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 1927.835333] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1934.112879] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1940.229271] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1948.129566] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1948.143773] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1948.158888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1948.646658] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1948.658849] Lustre: Skipped 1 previous similar message [ 1948.667955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1948.675027] Lustre: Skipped 1 previous similar message [ 1952.587456] Lustre: server umount lustre-MDT0000 complete [ 1956.519509] LustreError: 64639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787576569 with bad export cookie 9304303821467060220 [ 1956.521341] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1956.531697] LustreError: 64639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1956.876096] Lustre: server umount lustre-MDT0001 complete [ 1960.581308] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1966.070951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1974.244257] Lustre: 63556:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787576571/real 1787576571] req@000000004eba6d87 x1874409775211392/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787576587 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1974.272172] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1975.980833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1978.400143] Lustre: 63556:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787576575/real 1787576575] req@00000000229043b9 x1874409775211520/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787576591 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1980.384118] Lustre: 63557:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787576577/real 1787576577] req@00000000e8ca7a6e x1874409775211648/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787576593 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1981.481295] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x811f870780da90df [ 1981.503753] Lustre: MGC192.168.206.112@tcp: Connection restored to 0@lo (at 0@lo) [ 1981.996757] LustreError: 66497:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1982.023497] LustreError: 66497:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1982.095346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1982.198598] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1983.142436] Lustre: 63557:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787576579/real 1787576579] req@0000000086eb187b x1874409775211904/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787576595 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1983.148567] LustreError: 66498:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1983.184686] LustreError: 66498:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1984.162168] LustreError: 66497:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1984.183423] LustreError: 66497:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1987.549960] Lustre: 63556:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787576584/real 1787576584] req@00000000f2f12146 x1874409775212032/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787576600 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1987.692818] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 1989.601526] LustreError: 68419:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1989.630639] LustreError: 68419:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1994.734511] LustreError: 68780:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1994.781432] LustreError: 68780:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1998.034804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1998.499435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2003.630247] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2005.208601] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2015.720860] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2023.260837] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2034.154094] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2034.172183] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2034.192072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2039.779212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2039.784489] Lustre: Skipped 4 previous similar messages [ 2044.383155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2044.573946] Lustre: server umount lustre-MDT0000 complete [ 2044.913251] LustreError: 68417:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2044.947172] LustreError: 68417:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2048.491633] LustreError: 64640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787576661 with bad export cookie 9304303821467062495 [ 2048.501752] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2048.513561] LustreError: 64640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2048.985269] Lustre: server umount lustre-MDT0001 complete [ 2056.842645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2066.143289] Lustre: 63556:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787576663/real 1787576663] req@0000000032d8be39 x1874409775242240/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787576679 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2066.205361] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2066.234847] Lustre: Skipped 2 previous similar messages [ 2067.775111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2073.861688] LustreError: 66498:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2073.946118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2074.010736] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2074.995203] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2075.001801] Lustre: Skipped 1 previous similar message [ 2078.285463] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2086.797557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2092.524151] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2092.959060] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2100.634408] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2106.789640] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2118.127072] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2118.130847] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2118.146774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2118.160615] Lustre: Skipped 2 previous similar messages [ 2118.176511] Lustre: Skipped 2 previous similar messages [ 2121.076942] Lustre: server umount lustre-MDT0000 complete [ 2123.240945] LustreError: 66497:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2123.267040] LustreError: 66497:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 2125.307097] LustreError: 64640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787576738 with bad export cookie 9304303821467063874 [ 2125.321974] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2125.322843] LustreError: 64640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2125.783852] Lustre: server umount lustre-MDT0001 complete [ 2132.459295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2136.365218] Lustre: DEBUG MARKER: files compared the same [ 2149.161533] Lustre: server umount lustre-OST0000 complete [ 2169.289180] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 2173.066500] Key type lgssc unregistered [ 2173.338682] LNet: 74384:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2173.346994] LNetError: 74384:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2174.379812] LNet: Removed LNI 192.168.206.112@tcp [ 2175.449793] Key type .llcrypt unregistered [ 2175.454354] Key type ._llcrypt unregistered [ 2202.559276] Key type ._llcrypt registered [ 2202.563724] Key type .llcrypt registered [ 2202.721162] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 2204.269290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2204.304100] alg: No test for adler32 (adler32-zlib) [ 2205.734727] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [ 2206.107740] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 2207.831263] Key type lgssc registered [ 2209.533813] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2218.257242] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2218.281581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2219.910804] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2224.401959] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug +malloc all [ 2229.218234] LustreError: 75802:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2229.247797] LustreError: 75802:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2232.867808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2233.289674] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2237.667313] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug +malloc all [ 2245.207344] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2251.445419] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2257.683307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2258.091669] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2263.978761] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug +malloc all [ 2264.271863] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2272.293823] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2285.024633] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2285.025714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2285.050946] Lustre: Skipped 1 previous similar message [ 2285.065180] Lustre: Skipped 1 previous similar message [ 2286.761417] Lustre: server umount lustre-OST0000 complete [ 2294.752414] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2294.758270] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2294.783806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2297.362310] Lustre: server umount lustre-MDT0000 complete [ 2300.388534] LustreError: 75798:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2300.412140] LustreError: 75798:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2301.610783] LustreError: 75783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787576914 with bad export cookie 15065915826899680954 [ 2301.624164] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2301.626375] LustreError: 75783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2302.029555] Lustre: server umount lustre-MDT0001 complete [ 2313.531223] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 2316.926664] Key type lgssc unregistered [ 2317.311635] LNet: 79339:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2317.315406] LNetError: 79339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2317.325994] LNet: Removed LNI 192.168.206.112@tcp [ 2318.260482] Key type .llcrypt unregistered [ 2318.261942] Key type ._llcrypt unregistered [ 2341.847404] Key type ._llcrypt registered [ 2341.858336] Key type .llcrypt registered [ 2342.023549] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 2343.299467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2343.502063] alg: No test for adler32 (adler32-zlib) [ 2344.719281] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [ 2344.907587] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 2346.583158] Key type lgssc registered [ 2347.582616] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2355.128388] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2355.162678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2370.719522] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2375.839831] LustreError: 80731:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.112@tcp: failed processing log, type 4: rc = -110 [ 2402.593447] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2409.065837] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2416.175508] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2423.456355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2423.584428] Lustre: *** cfs_fail_loc=706, val=0*** [ 2423.982153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2424.059318] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2428.631823] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2429.408317] LustreError: 80756:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2429.440039] LustreError: 80756:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2434.533524] LustreError: 80757:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2437.849305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2438.217275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2443.049457] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2451.263841] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2457.286177] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2460.128389] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2460.131189] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2460.141798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2460.155142] Lustre: Skipped 1 previous similar message [ 2465.254142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2465.256530] Lustre: Skipped 1 previous similar message [ 2466.017708] Lustre: server umount lustre-OST0000 complete [ 2470.369655] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2470.387579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2470.393390] Lustre: Skipped 2 previous similar messages [ 2475.491144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2475.505713] Lustre: Skipped 1 previous similar message [ 2476.238831] Lustre: server umount lustre-MDT0000 complete [ 2479.811171] LustreError: 80737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787577092 with bad export cookie 8875825423855561886 [ 2479.826020] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2479.826701] LustreError: 80737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2480.180212] Lustre: server umount lustre-MDT0001 complete [ 2490.681207] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 2493.539820] Key type lgssc unregistered [ 2493.829697] LNet: 84352:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2493.843939] LNetError: 84352:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2493.867567] LNet: Removed LNI 192.168.206.112@tcp [ 2494.943184] Key type .llcrypt unregistered [ 2494.945138] Key type ._llcrypt unregistered [ 2526.880077] Key type ._llcrypt registered [ 2526.885398] Key type .llcrypt registered [ 2527.000430] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 2528.454474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2528.510679] alg: No test for adler32 (adler32-zlib) [ 2529.549676] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [ 2529.834467] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 2531.614618] Key type lgssc registered [ 2532.919479] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2541.912462] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2541.925502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2547.715550] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2557.525136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2558.210082] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2562.677604] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2563.559828] LustreError: 86457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2563.584063] LustreError: 86457:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2568.680356] LustreError: 86460:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2568.706237] LustreError: 86460:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2571.498833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2571.759325] LustreError: 86461:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2571.776520] LustreError: 86461:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2571.807643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2576.881877] LustreError: 86460:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2576.904410] LustreError: 86460:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2578.539918] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2581.986059] LustreError: 86456:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2582.009668] LustreError: 86456:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2587.848741] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2587.859501] Lustre: MGS: Client 17694304-e146-478a-9d2d-a708d98db618 (at 0@lo) reconnecting [ 2587.879143] LustreError: 86471:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2588.197032] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2588.307056] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2592.815396] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2608.618275] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2608.640833] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2608.647273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2608.661647] LustreError: Skipped 1 previous similar message [ 2608.713459] Lustre: Skipped 1 previous similar message [ 2612.455959] Lustre: server umount lustre-OST0000 complete [ 2618.848733] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2618.853175] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2618.861926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2618.883447] Lustre: Skipped 2 previous similar messages [ 2618.897306] Lustre: Skipped 1 previous similar message [ 2623.281346] Lustre: server umount lustre-MDT0000 complete [ 2623.969794] LustreError: 86456:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2623.977110] LustreError: 86456:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2627.302352] Lustre: server umount lustre-MDT0001 complete [ 2631.674731] Lustre: server umount MGS complete [ 2643.973081] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 09:14:15 (1787577255) [ 2673.202343] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [ 2684.402727] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 2723.555565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2728.184093] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2737.971821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2738.292579] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2738.343841] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2738.433210] Lustre: lustre-MDT0001: new disk, initializing [ 2738.522184] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2738.565906] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2738.577789] Lustre: 93625:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2738.600646] LustreError: 93625:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2743.551525] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2754.168054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2754.417523] Lustre: lustre-OST0000: new disk, initializing [ 2754.430887] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2754.444549] Lustre: 94391:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2754.516143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2761.037793] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2763.360903] Lustre: 94809:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000029059125 x1874410489290752/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787577392 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2770.565551] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2778.723202] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2778.751589] Lustre: MGS: Client fa4e4603-9c63-45c1-bff6-5f32f14044b0 (at 0@lo) reconnecting [ 2778.811657] Lustre: 95439:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2778.861571] LustreError: 93611:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2778.883530] LustreError: 93611:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2778.924523] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2779.038242] Lustre: lustre-MDT0000: new disk, initializing [ 2779.271608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2779.313551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2779.348781] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2780.387407] Lustre: 95463:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000068fe02b5 x1874410489297792/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787577409 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2784.223802] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2785.570306] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2789.493556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2789.517131] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2789.660605] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2816.482893] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2816.495210] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2816.498743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2816.522930] Lustre: Skipped 1 previous similar message [ 2821.602994] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2821.616642] Lustre: Skipped 1 previous similar message [ 2826.727044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2830.303676] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2830.492380] Lustre: server umount lustre-OST0000 complete [ 2836.962726] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2836.973256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2836.990230] Lustre: Skipped 2 previous similar messages [ 2837.010700] Lustre: Skipped 2 previous similar messages [ 2849.247938] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2849.439799] Lustre: server umount lustre-MDT0000 complete [ 2852.322768] LustreError: 93600:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2852.347858] LustreError: 93600:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2860.104700] Lustre: server umount lustre-MDT0001 complete [ 2864.444569] Lustre: server umount MGS complete [ 2876.508702] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 09:18:08 (1787577488) [ 2896.778245] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 2900.213988] Key type lgssc unregistered [ 2900.584539] LNet: 97968:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2900.590306] LNetError: 97968:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2900.613933] LNet: Removed LNI 192.168.206.112@tcp [ 2901.627738] Key type .llcrypt unregistered [ 2901.631057] Key type ._llcrypt unregistered [ 2920.955879] Key type ._llcrypt registered [ 2920.958204] Key type .llcrypt registered [ 2921.130399] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 2923.083760] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2923.175442] alg: No test for adler32 (adler32-zlib) [ 2924.367785] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [ 2924.824478] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 2926.583162] Key type lgssc registered [ 2928.207447] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2939.027625] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2939.089961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2941.015949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2946.394779] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2951.147523] LustreError: 99189:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2951.156913] LustreError: 99189:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2956.270484] LustreError: 99188:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2958.437299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2958.968919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2963.469369] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 2972.749439] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2979.213239] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2986.967240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2987.345447] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2995.509167] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 3002.483888] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3010.994955] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3021.285375] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3021.285498] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3021.293311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3021.300826] Lustre: Skipped 1 previous similar message [ 3026.422769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3026.439640] Lustre: Skipped 1 previous similar message [ 3026.874478] Lustre: server umount lustre-OST0000 complete [ 3030.496446] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3030.502270] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3030.515376] Lustre: Skipped 1 previous similar message [ 3030.522597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3031.521831] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3036.488444] Lustre: server umount lustre-MDT0000 complete [ 3036.651569] LustreError: 100285:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3036.672632] LustreError: 100285:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3040.684122] LustreError: 99168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787577653 with bad export cookie 8199302186611679264 [ 3040.690782] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3040.708399] LustreError: 99168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3041.140831] Lustre: server umount lustre-MDT0001 complete [ 3052.183818] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 3055.491417] Key type lgssc unregistered [ 3055.819212] LNet: 102747:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3055.832335] LNetError: 102747:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3055.853769] LNet: Removed LNI 192.168.206.112@tcp [ 3057.047599] Key type .llcrypt unregistered [ 3057.058909] Key type ._llcrypt unregistered [ 3076.425540] Key type ._llcrypt registered [ 3076.429603] Key type .llcrypt registered [ 3076.555609] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 3077.558723] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3077.616201] alg: No test for adler32 (adler32-zlib) [ 3078.862763] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [ 3079.164630] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 3080.935197] Key type lgssc registered [ 3082.126739] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3088.546600] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3088.554736] LustreError: 103830:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 3088.566266] LustreError: 103831:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 3088.570733] LustreError: 103830:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4992 [ 3088.581481] LustreError: 103830:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3088.586038] LustreError: 103830:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3088.592311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3090.106623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3098.077700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3098.566225] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3104.208782] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 3110.871586] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing lsmod [ 3113.745371] LustreError: 105034:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 3113.837760] LustreError: 105041:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 3113.845752] LustreError: 105034:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4916 [ 3113.856243] LustreError: 105034:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3113.865407] LustreError: 105034:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3113.870417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3114.292981] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3115.322551] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3119.072433] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3119.080385] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3119.096159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3120.615485] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3120.628816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3120.641038] Lustre: Skipped 1 previous similar message [ 3120.643406] Lustre: Skipped 1 previous similar message [ 3123.746840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3123.758012] Lustre: Skipped 3 previous similar messages [ 3128.804715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3128.812198] Lustre: Skipped 2 previous similar messages [ 3131.359357] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3131.509444] Lustre: server umount lustre-MDT0000 complete [ 3133.921680] LustreError: 104631:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3133.953309] LustreError: 104631:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3134.957717] LustreError: 103845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787577748 with bad export cookie 4264386499781474969 [ 3134.975616] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3134.979961] LustreError: 103845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3136.095486] LustreError: 104300:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000f12d456b x1874411065990912/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3136.326762] Lustre: server umount lustre-MDT0001 complete [ 3150.523964] Lustre: server umount lustre-OST0000 complete [ 3159.398902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3159.852067] LustreError: 106106:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3159.917995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3164.738495] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 3165.152970] LustreError: 106107:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3169.251415] LustreError: 106106:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3174.375296] LustreError: 106107:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3175.218367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3176.066932] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3182.440650] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 3193.139491] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3200.862266] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3208.220699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3208.821936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3215.579710] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 3223.945315] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3229.370727] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3234.788669] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3234.802758] Lustre: Skipped 1 previous similar message [ 3234.802840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3234.815224] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3234.827904] Lustre: Skipped 1 previous similar message [ 3240.769500] Lustre: server umount lustre-OST0000 complete [ 3245.027172] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3245.034097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3245.045981] Lustre: Skipped 1 previous similar message [ 3245.066596] Lustre: Skipped 3 previous similar messages [ 3250.706155] Lustre: server umount lustre-MDT0000 complete [ 3254.534455] LustreError: 106086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787577867 with bad export cookie 4264386499781475900 [ 3254.540822] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3254.543259] LustreError: 106086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3254.887651] Lustre: server umount lustre-MDT0001 complete [ 3269.373775] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 3274.294547] Key type lgssc unregistered [ 3274.756995] LNet: 109643:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3274.770633] LNetError: 109643:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3274.803428] LNet: Removed LNI 192.168.206.112@tcp [ 3276.129294] Key type .llcrypt unregistered [ 3276.131570] Key type ._llcrypt unregistered [ 3306.243306] Key type ._llcrypt registered [ 3306.249979] Key type .llcrypt registered [ 3306.364681] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 3307.770165] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3308.047449] alg: No test for adler32 (adler32-zlib) [ 3309.414507] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [ 3309.741590] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 3311.503167] Key type lgssc registered [ 3312.703730] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3321.304695] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3321.339303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3322.900267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3327.978591] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 3332.069728] LustreError: 111062:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3332.096324] LustreError: 111062:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3337.185855] LustreError: 111061:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3337.556711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3338.169463] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3344.087731] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 3354.578428] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3362.355820] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3371.522742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3371.899719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3379.391886] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 3388.157905] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3392.379803] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3398.287060] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3402.062534] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3407.844065] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3407.850743] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3407.857810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3407.867283] Lustre: Skipped 1 previous similar message [ 3412.402279] Lustre: server umount lustre-OST0000 complete [ 3418.081912] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3418.084032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3418.103442] Lustre: Skipped 1 previous similar message [ 3418.118319] Lustre: Skipped 2 previous similar messages [ 3423.202923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3423.208301] Lustre: Skipped 1 previous similar message [ 3428.321770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3428.329316] Lustre: Skipped 1 previous similar message [ 3430.367166] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3430.619663] Lustre: server umount lustre-MDT0000 complete [ 3433.446392] LustreError: 111988:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3433.475235] LustreError: 111988:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3434.786109] LustreError: 111041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787578047 with bad export cookie 11926994712393246257 [ 3434.795917] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3434.806696] LustreError: 111041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3435.613920] Lustre: server umount lustre-MDT0001 complete [ 3448.035767] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 3451.529981] Key type lgssc unregistered [ 3451.909705] LNet: 114711:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3451.921076] LNetError: 114711:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3451.951870] LNet: Removed LNI 192.168.206.112@tcp [ 3453.128296] Key type .llcrypt unregistered [ 3453.133481] Key type ._llcrypt unregistered [ 3475.039709] Key type ._llcrypt registered [ 3475.042491] Key type .llcrypt registered [ 3475.225178] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 3476.729554] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3476.793545] alg: No test for adler32 (adler32-zlib) [ 3477.992138] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [ 3478.183130] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 3479.807161] Key type lgssc registered [ 3481.137395] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3489.759657] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3489.775222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3491.414442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3496.057692] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 3501.538059] LustreError: 115934:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3501.569852] LustreError: 115934:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3506.346369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3506.657542] LustreError: 115935:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3506.888929] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3511.993610] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 3521.634926] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3527.889601] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3535.149493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3535.550803] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3536.557254] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3543.366943] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 3552.873245] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3562.472539] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3562.472573] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3562.484018] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3562.493643] Lustre: Skipped 1 previous similar message [ 3562.516906] Lustre: Skipped 1 previous similar message [ 3567.590561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3567.604122] Lustre: Skipped 1 previous similar message [ 3568.677854] Lustre: server umount lustre-OST0000 complete [ 3573.728817] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3573.742087] LustreError: Skipped 1 previous similar message [ 3573.748467] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3573.787535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3577.830196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3577.834981] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3580.408692] Lustre: server umount lustre-MDT0000 complete [ 3582.959190] LustreError: 115929:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3582.984613] LustreError: 115929:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3585.621730] LustreError: 115916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787578198 with bad export cookie 3272572826358758758 [ 3585.626843] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3585.655159] LustreError: 115916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3586.937791] Lustre: server umount lustre-MDT0001 complete [ 3605.169290] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 3609.271076] Key type lgssc unregistered [ 3609.574309] LNet: 119493:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3609.585429] LNetError: 119493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3609.609553] LNet: Removed LNI 192.168.206.112@tcp [ 3610.930192] Key type .llcrypt unregistered [ 3610.934337] Key type ._llcrypt unregistered [ 3638.581795] Key type ._llcrypt registered [ 3638.587607] Key type .llcrypt registered [ 3638.728119] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 3640.281100] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3640.645335] alg: No test for adler32 (adler32-zlib) [ 3641.858561] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [ 3642.130860] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 3643.816446] Key type lgssc registered [ 3645.374804] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3655.127936] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3655.177915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3656.631153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3663.826345] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 3665.904390] LustreError: 120910:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3665.915879] LustreError: 120910:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3671.012729] LustreError: 120911:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3676.130845] LustreError: 120910:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3677.810290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3678.893790] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3685.647483] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 3694.105845] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3701.471321] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3710.313039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3710.793939] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3718.424931] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 3727.163289] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3727.374518] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3735.570265] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3740.052381] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3740.056658] Lustre: Skipped 2 previous similar messages [ 3743.816222] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3743.843064] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3744.933076] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3744.939313] Lustre: Skipped 2 previous similar messages [ 3752.012428] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3752.019615] Lustre: Skipped 1 previous similar message [ 3756.989721] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.206.12@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3756.998065] Lustre: Skipped 2 previous similar messages [ 3760.154533] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.206.12@tcp 192.168.0.[2,10]@tcp [ 3770.553994] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.206.12/24@tcp in log lustre-MDT0000 [ 3770.559380] Lustre: Skipped 2 previous similar messages [ 3777.642451] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.12/24@tcp [ 3777.647768] Lustre: Skipped 1 previous similar message [ 3783.648246] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3783.665233] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3783.666220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3783.674245] Lustre: Skipped 1 previous similar message [ 3783.701391] Lustre: Skipped 1 previous similar message [ 3785.354911] Lustre: server umount lustre-OST0000 complete [ 3791.328233] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3791.335467] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3791.354916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3793.904211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3793.905044] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3799.013431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3799.027648] Lustre: Skipped 2 previous similar messages [ 3803.105175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3803.110932] Lustre: Skipped 1 previous similar message [ 3804.127122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3804.362390] Lustre: server umount lustre-MDT0000 complete [ 3808.068609] LustreError: 120893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787578421 with bad export cookie 17067143530462359422 [ 3808.069176] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3808.081689] LustreError: 120893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3808.433342] Lustre: server umount lustre-MDT0001 complete [ 3819.613773] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 3822.325838] Key type lgssc unregistered [ 3822.678724] LNet: 125814:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3822.687768] LNetError: 125814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3823.718955] LNet: Removed LNI 192.168.206.112@tcp [ 3824.803313] Key type .llcrypt unregistered [ 3824.806382] Key type ._llcrypt unregistered [ 3852.315979] Key type ._llcrypt registered [ 3852.318425] Key type .llcrypt registered [ 3852.413652] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 3853.445465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3853.615465] alg: No test for adler32 (adler32-zlib) [ 3854.829494] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [ 3855.103485] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 3856.807283] Key type lgssc registered [ 3858.013872] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3865.443054] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3865.491904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3867.212115] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3867.258244] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.206.12/24@tcp [ 3867.259096] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3872.119877] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 3877.353882] LustreError: 127373:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3877.400440] LustreError: 127373:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3882.478736] LustreError: 127374:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3884.291470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3885.177360] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3885.207573] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.12/24@tcp [ 3885.323374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3891.869598] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 3903.298540] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3909.915712] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3917.273525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3917.794053] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3923.526382] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 3931.295878] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3937.277119] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 3955.697103] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3955.706341] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3955.745330] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3957.728099] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3957.749409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3957.753050] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3957.824268] Lustre: Skipped 1 previous similar message [ 3960.415581] Lustre: server umount lustre-OST0000 complete [ 3966.945985] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3966.962744] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3966.980036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3971.978299] Lustre: server umount lustre-MDT0000 complete [ 3973.093504] LustreError: 127368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3973.112469] LustreError: 127368:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3976.070982] LustreError: 127356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787578589 with bad export cookie 8957185904920174871 [ 3976.094415] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3976.476179] Lustre: server umount lustre-MDT0001 complete [ 3986.859260] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 3990.069064] Key type lgssc unregistered [ 3990.599088] LNet: 130986:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3990.617801] LNetError: 130986:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3990.657484] LNet: Removed LNI 192.168.206.112@tcp [ 3991.757362] Key type .llcrypt unregistered [ 3991.759548] Key type ._llcrypt unregistered [ 4028.181302] Key type ._llcrypt registered [ 4028.198513] Key type .llcrypt registered [ 4028.450278] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [ 4044.205484] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 4045.914222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4046.034669] alg: No test for adler32 (adler32-zlib) [ 4047.253514] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [ 4047.508221] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 4049.239165] Key type lgssc registered [ 4050.387344] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4093.615888] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4093.632875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4094.785376] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4094.884440] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4095.022951] Lustre: lustre-MDT0000: new disk, initializing [ 4095.215073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4095.272322] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4101.127225] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 4112.910949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4113.130513] Lustre: 134390:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4113.221316] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4113.228364] Lustre: Skipped 1 previous similar message [ 4113.364422] Lustre: lustre-MDT0001: new disk, initializing [ 4113.547698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4113.585386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4113.593941] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4118.539873] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 4132.835978] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4143.360198] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4155.833045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4156.516050] Lustre: lustre-OST0000: new disk, initializing [ 4156.535887] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4156.558996] Lustre: 136361:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4156.765648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4162.250710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4162.269669] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4162.471048] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4165.804054] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 4176.651493] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4182.800540] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4183.167827] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4189.858878] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4190.241290] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4193.249380] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4193.250523] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4193.263876] LustreError: Skipped 1 previous similar message [ 4193.289099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4193.294463] Lustre: Skipped 1 previous similar message [ 4193.317423] Lustre: Skipped 1 previous similar message [ 4198.373292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4198.386013] Lustre: Skipped 1 previous similar message [ 4199.231814] Lustre: server umount lustre-OST0000 complete [ 4205.536623] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4205.562232] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4205.593843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4208.608868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4208.609374] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4210.836568] Lustre: server umount lustre-MDT0000 complete [ 4215.194432] LustreError: 134383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787578828 with bad export cookie 17555314738970765217 [ 4215.196296] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4215.217616] LustreError: 134383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4215.609478] Lustre: server umount lustre-MDT0001 complete [ 4225.721423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4226.430511] LustreError: 138587:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4226.451457] LustreError: 138587:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4226.540718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4231.308516] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 4231.653269] LustreError: 138588:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4236.773868] LustreError: 138587:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4241.140843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4241.543754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4246.955977] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 4255.628514] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4261.753150] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4269.133340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4269.460175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4275.649291] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 4282.244233] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4287.123741] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4287.336644] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4292.344280] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4292.537381] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4302.703897] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4302.993904] Lustre: lustre-OST0001: new disk, initializing [ 4303.003693] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4303.013047] Lustre: 141961:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4303.168396] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4310.711455] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 4312.250628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4312.262484] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4312.418421] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4320.102690] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4320.297314] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4327.156459] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4327.508808] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4334.661372] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8a6307798800.ost_server_uuid 50 [ 4337.124255] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8a6307798800.ost_server_uuid in IDLE state after 0 sec [ 4348.397389] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4348.409323] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4348.431707] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4348.437188] Lustre: Skipped 2 previous similar messages [ 4350.227969] Lustre: server umount lustre-OST0001 complete [ 4353.517881] LustreError: 141281:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4353.558976] LustreError: 141281:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4358.625108] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4358.630513] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4358.633248] LustreError: Skipped 2 previous similar messages [ 4358.641293] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4358.662839] Lustre: Skipped 2 previous similar messages [ 4358.681455] Lustre: Skipped 1 previous similar message [ 4358.682690] LustreError: 140412:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4358.699813] LustreError: 140412:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4360.634320] Lustre: server umount lustre-OST0000 complete [ 4368.871655] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4368.887659] Lustre: Skipped 1 previous similar message [ 4370.878725] Lustre: server umount lustre-MDT0000 complete [ 4373.990491] LustreError: 139315:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4374.007587] LustreError: 139315:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4374.774603] LustreError: 138569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787578987 with bad export cookie 17555314738970766092 [ 4374.782295] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4374.788394] LustreError: 138569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4375.184286] Lustre: server umount lustre-MDT0001 complete [ 4395.331530] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 4398.301395] Key type lgssc unregistered [ 4398.614846] LNet: 144968:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4398.629691] LNetError: 144968:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4398.650314] LNet: Removed LNI 192.168.206.112@tcp [ 4399.507858] Key type .llcrypt unregistered [ 4399.512386] Key type ._llcrypt unregistered [ 4444.304886] Key type ._llcrypt registered [ 4444.308463] Key type .llcrypt registered [ 4444.461677] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [ 4462.784883] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 4464.515676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4464.572476] alg: No test for adler32 (adler32-zlib) [ 4465.962414] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [ 4466.269489] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 4467.983189] Key type lgssc registered [ 4469.243428] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4519.279287] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4519.342927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4520.884249] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4520.980059] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4521.117159] Lustre: lustre-MDT0000: new disk, initializing [ 4521.253795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4521.294942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4527.023449] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 4537.719329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4537.883554] Lustre: 149129:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4537.978379] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4537.986228] Lustre: Skipped 1 previous similar message [ 4538.103460] Lustre: lustre-MDT0001: new disk, initializing [ 4538.233980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4538.292821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4538.324926] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4543.050704] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 4555.800888] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4564.834616] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4574.133042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4574.539163] Lustre: lustre-OST0000: new disk, initializing [ 4574.545427] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4574.557592] Lustre: 151099:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4574.702795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4583.624432] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4583.640805] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4583.901560] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4584.237567] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 4595.944278] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4601.306086] Lustre: Failing over lustre-OST0000 [ 4601.506457] Lustre: server umount lustre-OST0000 complete [ 4617.705384] Lustre: 147353:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787579214/real 1787579214] req@000000001383e0dc x1874412520377984/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1787579230 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4617.755069] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4620.895082] Lustre: 147350:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787579217/real 1787579217] req@00000000a84b9da1 x1874412520379136/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787579233 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4620.937816] Lustre: 147350:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4620.949239] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4622.818591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4623.087293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4623.102464] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4625.128499] Lustre: 147350:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787579222/real 1787579222] req@00000000c7587d8d x1874412520380160/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787579238 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4625.199875] Lustre: 147350:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4627.173742] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4630.242468] Lustre: 147351:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787579227/real 1787579227] req@000000000beb30df x1874412520381184/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787579243 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4631.356049] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4631.356187] Lustre: lustre-OST0000: Recovery over after 0:04, of 3 clients 3 recovered and 0 were evicted. [ 4631.779066] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 4636.247102] Lustre: Failing over lustre-MDT0000 [ 4636.755249] Lustre: server umount lustre-MDT0000 complete [ 4637.680457] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4637.710312] Lustre: Skipped 2 previous similar messages [ 4642.789350] LustreError: 149138:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4642.824910] LustreError: 149138:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4644.802577] LustreError: 149166:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4647.908677] LustreError: 149136:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4647.925952] LustreError: 149136:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4649.931109] LustreError: 149166:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4653.026311] Lustre: 147352:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787579250/real 1787579250] req@000000009f62f935 x1874412520394368/t0(0) o400->MGC192.168.206.112@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787579266 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4653.063967] Lustre: 147352:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4653.073131] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4655.048921] LustreError: 151107:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4655.071774] LustreError: 151107:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4656.594482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4663.274105] LustreError: 149136:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4663.304039] LustreError: 149136:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 4663.613691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4663.678892] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4666.306581] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4668.912755] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4668.928076] Lustre: Skipped 1 previous similar message [ 4668.960531] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 4669.000320] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4669.157933] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 4679.140682] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4679.147608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4679.160293] Lustre: Skipped 1 previous similar message [ 4684.267071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4684.293612] Lustre: Skipped 2 previous similar messages [ 4684.714069] Lustre: server umount lustre-OST0000 complete [ 4689.377743] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4689.379113] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4689.394552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4689.421584] Lustre: Skipped 1 previous similar message [ 4694.502685] LustreError: 149141:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4694.524364] LustreError: 149141:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4696.201598] Lustre: server umount lustre-MDT0000 complete [ 4699.641881] LustreError: 151098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787579312 with bad export cookie 15487593511922387422 [ 4699.652030] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4699.667944] LustreError: 151098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4700.103434] Lustre: server umount lustre-MDT0001 complete [ 4712.068348] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 4714.790718] Key type lgssc unregistered [ 4715.028090] LNet: 154759:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4715.038250] LNetError: 154759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4715.059339] LNet: Removed LNI 192.168.206.112@tcp [ 4715.823925] Key type .llcrypt unregistered [ 4715.826148] Key type ._llcrypt unregistered [ 4754.234091] Key type ._llcrypt registered [ 4754.237218] Key type .llcrypt registered [ 4754.410106] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 4755.564716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4755.780701] alg: No test for adler32 (adler32-zlib) [ 4756.944197] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [ 4757.269691] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 4759.047527] Key type lgssc registered [ 4760.310134] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4767.950343] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4767.971136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4769.487303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4773.659327] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 4778.990874] LustreError: 156280:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4779.028988] LustreError: 156280:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4784.049188] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4784.108321] LustreError: 156281:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4789.220841] LustreError: 156280:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4789.257587] LustreError: 156309:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4789.264593] LustreError: 156309:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4789.274731] LustreError: 156309:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 4789.547924] Lustre: server umount lustre-MDT0000 complete [ 4807.323450] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [ 4816.455951] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 4851.531478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4851.860570] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4851.905667] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4852.028904] Lustre: lustre-MDT0000: new disk, initializing [ 4852.122983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4852.137084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4857.561339] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 4869.193046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4869.285781] Lustre: 159613:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4869.332499] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4869.337070] Lustre: Skipped 1 previous similar message [ 4869.404826] Lustre: lustre-MDT0001: new disk, initializing [ 4869.536955] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4869.598968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4869.606882] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4874.840990] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 4886.567680] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4893.257700] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4901.617439] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4902.225619] Lustre: lustre-OST0000: new disk, initializing [ 4902.232127] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4902.251072] Lustre: 161582:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4902.424965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4911.228885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4911.249937] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4911.452085] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4911.638468] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 4925.300547] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4930.263926] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4932.748891] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 09:52:24 (1787579544) [ 5198.304446] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5198.307728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5198.326052] Lustre: Skipped 2 previous similar messages [ 5198.343761] Lustre: Skipped 2 previous similar messages [ 5203.424308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5203.443848] Lustre: Skipped 2 previous similar messages [ 5203.845181] Lustre: server umount lustre-MDT0000 complete [ 5207.425563] LustreError: 159606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787579820 with bad export cookie 1365441383151412257 [ 5207.437410] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5207.448182] LustreError: 159606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5208.544365] LustreError: 159620:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5208.545480] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5208.593251] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5212.577510] LustreError: 159619:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5212.580982] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5213.917672] Lustre: server umount lustre-MDT0001 complete [ 5224.772755] Lustre: server umount lustre-OST0000 complete [ 5234.808866] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [ 5244.905870] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 5281.583303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5281.952736] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5281.996650] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5282.093891] Lustre: lustre-MDT0000: new disk, initializing [ 5282.197992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5282.241132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5286.550833] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 5297.295385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5297.453790] Lustre: 166251:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5297.545485] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5297.556464] Lustre: Skipped 1 previous similar message [ 5297.701486] Lustre: lustre-MDT0001: new disk, initializing [ 5297.796296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5297.840119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5297.860401] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5302.966294] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 5312.877023] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5319.155770] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5326.896033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5327.198293] Lustre: lustre-OST0000: new disk, initializing [ 5327.204668] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5327.213427] Lustre: 168219:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5327.315658] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5333.669807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5333.687845] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5333.915780] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5335.514380] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 5345.747232] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5350.682430] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5350.887231] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5355.503752] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5355.709718] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5359.601937] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5359.614170] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5359.621304] LustreError: Skipped 1 previous similar message [ 5359.624608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5359.670229] Lustre: Skipped 1 previous similar message [ 5369.835185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5369.848277] Lustre: Skipped 4 previous similar messages [ 5372.383141] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5372.555899] Lustre: server umount lustre-OST0000 complete [ 5380.067920] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5380.071126] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5380.095799] Lustre: Skipped 1 previous similar message [ 5382.873659] Lustre: server umount lustre-MDT0000 complete [ 5385.185308] LustreError: 167667:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5385.198742] LustreError: 167667:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5386.777331] LustreError: 166241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787579999 with bad export cookie 1365441383151571087 [ 5386.778520] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5386.788830] LustreError: 166241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5387.180282] Lustre: server umount lustre-MDT0001 complete [ 5396.321850] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:00:08 (1787580008) [ 5414.303924] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [ 5424.626889] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 5461.998592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5462.154811] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5462.187881] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5462.263955] Lustre: lustre-MDT0000: new disk, initializing [ 5462.356219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5462.376480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5467.005333] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 5478.122324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5478.283874] Lustre: 172992:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5478.590219] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5483.772365] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 5494.127453] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5499.321198] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5505.769808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5506.046133] Lustre: lustre-OST0000: new disk, initializing [ 5506.051240] Lustre: Skipped 1 previous similar message [ 5506.061096] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5506.070508] Lustre: Skipped 2 previous similar messages [ 5506.075896] Lustre: 174968:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5511.206786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5511.214029] Lustre: Skipped 1 previous similar message [ 5511.220192] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5511.336764] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5512.601558] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 5522.490026] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5531.341757] Lustre: server umount lustre-OST0000 complete [ 5537.249152] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5537.257559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5537.265829] Lustre: Skipped 1 previous similar message [ 5537.277543] Lustre: Skipped 2 previous similar messages [ 5545.627961] Lustre: server umount lustre-MDT0000 complete [ 5547.503195] LustreError: 172997:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5547.528826] LustreError: 172997:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5548.511154] Lustre: 155591:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787580145/real 1787580145] req@000000007b3f0171 x1874412826257664/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787580161 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5548.540305] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5549.553837] LustreError: 173801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787580162 with bad export cookie 1365441383151571962 [ 5549.554792] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5549.562727] LustreError: 173801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5549.902718] Lustre: server umount lustre-MDT0001 complete [ 5558.513255] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:02:50 (1787580170) [ 5575.823818] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [ 5586.827455] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 5627.255403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5627.682235] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5627.735526] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5627.851182] Lustre: lustre-MDT0000: new disk, initializing [ 5628.001323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5628.009366] Lustre: Skipped 2 previous similar messages [ 5628.046975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5633.231692] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 5645.855515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5645.990882] Lustre: 179294:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5646.268294] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5650.799417] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 5660.822335] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5666.974759] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5674.695591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5674.995659] Lustre: 181270:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5681.526633] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 5682.770611] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5682.917133] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5691.728665] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5703.658283] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5703.668266] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5703.682561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5703.686960] Lustre: Skipped 4 previous similar messages [ 5713.375916] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5713.590790] Lustre: server umount lustre-OST0000 complete [ 5717.990573] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5717.997375] LustreError: Skipped 1 previous similar message [ 5724.034397] Lustre: server umount lustre-MDT0000 complete [ 5724.128963] LustreError: 179304:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5724.155801] LustreError: 179304:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5727.725739] LustreError: 179285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787580340 with bad export cookie 1365441383151573054 [ 5727.730763] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5727.734931] LustreError: 179285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5728.144714] Lustre: server umount lustre-MDT0001 complete [ 5740.932810] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 5743.639926] Key type lgssc unregistered [ 5743.955854] LNet: 183134:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5743.967265] LNetError: 183134:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5743.994787] LNet: Removed LNI 192.168.206.112@tcp [ 5744.821958] Key type .llcrypt unregistered [ 5744.825817] Key type ._llcrypt unregistered [ 5769.996733] Key type ._llcrypt registered [ 5770.000219] Key type .llcrypt registered [ 5770.103801] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 5771.257229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5771.526125] alg: No test for adler32 (adler32-zlib) [ 5772.610393] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [ 5772.910782] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 5774.655877] Key type lgssc registered [ 5775.937692] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5784.770562] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5784.787666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5786.327829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5791.434799] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 5796.843255] LustreError: 184554:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5796.863837] LustreError: 184554:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5797.857879] LustreError: 184553:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5801.682166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5802.353028] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5807.037634] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 5814.985847] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5821.062992] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5828.850900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5829.509332] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5832.570358] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5836.227367] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 5844.458145] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5868.520846] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5868.524698] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5868.538960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5868.571375] Lustre: Skipped 1 previous similar message [ 5868.591098] Lustre: Skipped 1 previous similar message [ 5873.115437] Lustre: server umount lustre-OST0000 complete [ 5878.757050] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5878.783724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5878.788275] Lustre: Skipped 1 previous similar message [ 5883.846906] Lustre: server umount lustre-MDT0000 complete [ 5883.878438] LustreError: 184548:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5883.903079] LustreError: 184548:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5887.655317] LustreError: 184534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787580500 with bad export cookie 14526139116644592929 [ 5887.660797] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5887.668326] LustreError: 184534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5888.007438] Lustre: server umount lustre-MDT0001 complete [ 5899.286523] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 5902.217735] Key type lgssc unregistered [ 5902.563064] LNet: 188114:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5902.573268] LNetError: 188114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5902.617690] LNet: Removed LNI 192.168.206.112@tcp [ 5903.738405] Key type .llcrypt unregistered [ 5903.740459] Key type ._llcrypt unregistered [ 5929.382475] Key type ._llcrypt registered [ 5929.384978] Key type .llcrypt registered [ 5929.510576] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 5930.906733] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5931.009135] alg: No test for adler32 (adler32-zlib) [ 5932.064240] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [ 5932.330934] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 5934.015845] Key type lgssc registered [ 5935.311880] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5943.890664] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5943.919261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5945.505564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5950.647393] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 5955.045077] LustreError: 189535:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5955.074747] LustreError: 189535:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5960.161526] LustreError: 189534:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5960.355612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5960.789151] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5965.556028] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 5973.840504] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5979.305333] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5986.355100] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5986.761268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5990.839674] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5993.056242] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6000.441368] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6010.336328] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6010.341431] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6010.366108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6010.373854] Lustre: Skipped 1 previous similar message [ 6011.663892] Lustre: server umount lustre-OST0000 complete [ 6019.637607] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8a63050ca800.ost_server_uuid 50 [ 6029.955648] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6046.179990] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6046.198835] Lustre: Skipped 1 previous similar message [ 6046.212734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6046.224570] Lustre: Skipped 1 previous similar message [ 6049.985042] Lustre: server umount lustre-MDT0000 complete [ 6050.337100] LustreError: 189530:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6050.347281] LustreError: 189530:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6053.836287] LustreError: 192484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787580666 with bad export cookie 16897441188821322203 [ 6053.838745] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6053.853760] LustreError: 192484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6054.134462] Lustre: server umount lustre-MDT0001 complete [ 6063.145328] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 10:11:14 (1787580674) [ 6070.761035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6071.319039] LustreError: 193454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6071.398194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6076.387418] LustreError: 193455:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6076.662366] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6085.641961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6086.102151] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6090.880938] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6099.566359] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6105.674817] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6114.595610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6115.097039] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6116.156772] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6121.690776] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6130.358392] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6137.921384] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6138.229212] Lustre: lustre-OST0001: new disk, initializing [ 6138.235538] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6138.249335] Lustre: 196296:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6138.412598] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6145.764975] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6147.827103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6147.833704] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6147.990562] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6156.089332] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6161.896493] Lustre: lustre-OST0000: Not available for connect from 192.168.206.12@tcp (stopping) [ 6162.401348] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6162.409559] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6162.414755] LustreError: Skipped 1 previous similar message [ 6162.438449] Lustre: Skipped 2 previous similar messages [ 6165.519467] Lustre: server umount lustre-OST0000 complete [ 6166.994504] LustreError: 195278:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6167.036210] LustreError: 195278:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6173.249800] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6173.462385] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6179.055151] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6179.313257] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6183.913243] LustreError: 195277:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6183.942079] LustreError: 195277:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 6194.153988] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6194.162980] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6194.181388] Lustre: Skipped 1 previous similar message [ 6194.190500] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6194.195944] Lustre: Skipped 4 previous similar messages [ 6199.274818] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6199.279892] Lustre: Skipped 2 previous similar messages [ 6199.541166] Lustre: server umount lustre-OST0001 complete [ 6203.874807] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6203.887848] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6209.415527] Lustre: server umount lustre-MDT0000 complete [ 6212.558166] LustreError: 193435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787580825 with bad export cookie 16897441188821323274 [ 6212.568229] LustreError: 193435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6212.568395] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6213.060457] Lustre: server umount lustre-MDT0001 complete [ 6234.569669] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 10:14:06 (1787580846) [ 6241.302599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6241.552829] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6241.592898] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6241.775623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6246.556516] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6256.211804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6256.286343] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6256.324179] Lustre: 199733:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6262.249324] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6269.903486] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6276.469564] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6283.677826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6283.932515] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6290.566281] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6291.107070] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6299.278395] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6305.675203] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6305.773986] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6305.971356] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6305.982212] Lustre: Skipped 2 previous similar messages [ 6311.462757] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6314.159596] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6318.705819] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6321.636403] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6321.645307] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6321.665784] Lustre: Skipped 1 previous similar message [ 6321.670850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6321.676631] Lustre: Skipped 3 previous similar messages [ 6327.615483] Lustre: server umount lustre-OST0000 complete [ 6329.818952] LustreError: 201586:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6329.842501] LustreError: 201586:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 6344.672440] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6344.680970] LustreError: Skipped 1 previous similar message [ 6344.684499] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6344.696370] Lustre: Skipped 1 previous similar message [ 6344.701668] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6344.709163] Lustre: Skipped 4 previous similar messages [ 6350.098429] Lustre: server umount lustre-OST0001 complete [ 6368.223421] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6368.451297] Lustre: server umount lustre-MDT0000 complete [ 6372.332672] LustreError: 199726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787580985 with bad export cookie 16897441188821324674 [ 6372.339235] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6372.344376] LustreError: 199726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6372.674230] Lustre: server umount lustre-MDT0001 complete [ 6399.994793] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 10:16:51 (1787581011) [ 6419.634334] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [ 6434.244827] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 6474.933538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6475.177822] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6475.200941] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6475.261624] Lustre: lustre-MDT0000: new disk, initializing [ 6475.322925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6475.339816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6481.395962] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6493.000294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6493.264667] Lustre: 207856:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6493.319526] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6493.324819] Lustre: Skipped 1 previous similar message [ 6493.449843] Lustre: lustre-MDT0001: new disk, initializing [ 6493.634602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6493.651288] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6498.349364] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6509.178868] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6517.889414] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6525.713106] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6525.926390] Lustre: lustre-OST0000: new disk, initializing [ 6525.930333] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6525.936504] Lustre: 209824:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6532.706823] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6533.707929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6533.720969] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6533.930230] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6543.902934] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6549.514516] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6549.710622] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6556.122930] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6556.485252] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6559.722317] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6559.734225] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6559.740467] LustreError: Skipped 3 previous similar messages [ 6559.748805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6559.765516] Lustre: Skipped 4 previous similar messages [ 6559.795241] Lustre: Skipped 10 previous similar messages [ 6565.501125] Lustre: server umount lustre-OST0000 complete [ 6584.287807] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6584.716285] Lustre: server umount lustre-MDT0000 complete [ 6585.314185] LustreError: 207867:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6585.322277] LustreError: 207867:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 6589.390077] LustreError: 209835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787581202 with bad export cookie 16897441188821326466 [ 6589.408834] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6589.413414] LustreError: 209835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6589.769845] Lustre: server umount lustre-MDT0001 complete [ 6601.536136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6609.475386] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6619.268940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6625.634880] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6634.541980] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6641.064710] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6648.649158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6655.983708] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6664.642695] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6671.355478] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6673.954052] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 6679.816090] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6680.085781] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6683.104577] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6683.124078] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6683.139234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6683.166052] Lustre: Skipped 3 previous similar messages [ 6683.199701] Lustre: Skipped 9 previous similar messages [ 6688.746634] Lustre: server umount lustre-OST0000 complete [ 6697.989477] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6703.423566] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6733.406964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6733.979641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6733.989127] Lustre: Skipped 5 previous similar messages [ 6738.087939] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6738.105717] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6738.107302] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6738.132332] Lustre: Skipped 1 previous similar message [ 6740.467361] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6747.993628] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6751.286121] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 6758.886256] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6762.748815] Lustre: server umount lustre-OST0000 complete [ 6772.744227] Lustre: server umount lustre-MDT0000 complete [ 6774.241319] LustreError: 212046:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6774.263391] LustreError: 212046:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6776.768830] LustreError: 212029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787581389 with bad export cookie 16897441188821327348 [ 6776.775848] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6776.785651] LustreError: 212029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6777.182093] Lustre: server umount lustre-MDT0001 complete [ 6785.527534] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 10:23:17 (1787581397) [ 6793.621746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6799.846916] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6810.190944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6816.110821] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6825.502936] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6832.915211] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6841.854249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6850.538460] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6860.089718] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6864.926728] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6865.223711] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6869.675404] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6869.942494] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6877.052057] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6877.322474] Lustre: lustre-OST0001: new disk, initializing [ 6877.331082] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6877.344994] Lustre: 221405:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6884.530481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6884.540116] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6884.714838] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6885.343981] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6896.207118] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6901.469927] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6901.736599] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6906.414701] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6906.684656] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6910.432259] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6910.440396] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6910.451269] LustreError: Skipped 2 previous similar messages [ 6910.465808] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6910.474121] Lustre: Skipped 5 previous similar messages [ 6910.504322] Lustre: Skipped 8 previous similar messages [ 6923.231254] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6923.602064] Lustre: server umount lustre-OST0001 complete [ 6931.578503] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6931.860367] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6937.866735] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6938.082336] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6968.407678] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6973.941716] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6973.969665] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6973.970509] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6975.868411] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 6984.816756] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6986.778641] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6994.616697] Lustre: server umount lustre-OST0001 complete [ 7017.290592] Lustre: server umount lustre-OST0000 complete [ 7036.383170] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7036.858688] Lustre: server umount lustre-MDT0000 complete [ 7040.482030] LustreError: 218034:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7040.516310] LustreError: 218034:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 37 previous similar messages [ 7041.553106] LustreError: 223768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787581654 with bad export cookie 16897441188821328314 [ 7041.557372] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7041.564454] LustreError: 223768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7048.232386] Lustre: server umount lustre-MDT0001 complete [ 7073.062795] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 10:28:04 (1787581684) [ 7080.125229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7080.455609] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7080.484992] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7084.994063] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 7093.630032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7093.716551] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7093.755139] Lustre: 226652:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7099.742651] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 7107.806318] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7113.997653] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7120.477147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7120.645759] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7128.063285] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 7135.700722] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7145.485532] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7145.538189] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7152.810276] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 7160.794173] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7162.378390] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7167.007211] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7167.203690] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7171.933889] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7172.130725] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7178.330561] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8a6302c1e800.ost_server_uuid 50 [ 7179.992986] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8a6302c1e800.ost_server_uuid in IDLE state after 0 sec [ 7181.143175] Lustre: Permanently deactivating lustre-OST0001 [ 7181.161679] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7185.268484] Lustre: Permanently reactivating lustre-OST0001 [ 7185.308968] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7185.324426] Lustre: Skipped 2 previous similar messages [ 7195.104751] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7195.111611] LustreError: Skipped 3 previous similar messages [ 7195.122272] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7195.135691] Lustre: Skipped 6 previous similar messages [ 7195.155080] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7195.167586] Lustre: Skipped 19 previous similar messages [ 7197.395107] Lustre: server umount lustre-OST0001 complete [ 7208.123524] Lustre: server umount lustre-OST0000 complete [ 7218.639121] Lustre: server umount lustre-MDT0000 complete [ 7222.606432] LustreError: 226643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787581835 with bad export cookie 16897441188821329602 [ 7222.612785] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7222.623072] LustreError: 226643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7223.085973] Lustre: server umount lustre-MDT0001 complete [ 7248.935932] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 10:31:00 (1787581860) [ 7258.434996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7259.044574] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7259.118492] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7259.130956] Lustre: Skipped 2 previous similar messages [ 7259.538389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7259.551963] Lustre: Skipped 9 previous similar messages [ 7266.536722] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 7279.620887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7279.749615] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7279.807897] Lustre: 233346:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7287.099098] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 7295.854838] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7301.912706] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7308.829679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7309.053938] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7309.094974] Lustre: Permanently deactivating lustre-OST0000 [ 7309.112525] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7315.461473] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 7319.111031] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7324.598752] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7331.884120] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7331.967791] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7337.915580] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 7340.268946] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7346.158510] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7348.074809] Lustre: Permanently reactivating lustre-OST0000 [ 7348.093421] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7348.101392] Lustre: Skipped 2 previous similar messages [ 7354.958587] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 7354.970945] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 36 seconds. I think it's dead, and I am evicting it. exp 00000000b6a9a863, cur 1787581968 deadline 1787581962 last 1787581932 [ 7355.082433] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7361.737567] Lustre: server umount lustre-OST0001 complete [ 7367.166184] Lustre: server umount lustre-OST0000 complete [ 7373.185277] Lustre: server umount lustre-MDT0000 complete [ 7376.806191] LustreError: 235215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787581989 with bad export cookie 16897441188821331275 [ 7376.807931] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7376.823613] LustreError: 235215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7377.418306] Lustre: server umount lustre-MDT0001 complete [ 7389.193114] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 7392.954133] Key type lgssc unregistered [ 7393.309230] LNet: 238203:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7393.321608] LNetError: 238203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7394.343978] LNet: Removed LNI 192.168.206.112@tcp [ 7395.414576] Key type .llcrypt unregistered [ 7395.419250] Key type ._llcrypt unregistered [ 7419.760341] Key type ._llcrypt registered [ 7419.762664] Key type .llcrypt registered [ 7419.848151] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 7421.144617] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7421.160427] alg: No test for adler32 (adler32-zlib) [ 7422.252934] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [ 7422.481231] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 7424.159599] Key type lgssc registered [ 7425.284351] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7435.334752] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7435.357950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7436.875082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7442.322109] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 7445.997927] LustreError: 239671:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7446.020667] LustreError: 239671:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7451.104843] LustreError: 239672:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7451.999485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7452.150493] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7452.192237] Lustre: 239660:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7452.200082] Lustre: Permanently deactivating lustre-MDT0001 [ 7452.213890] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7452.497818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7457.566713] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 7460.388746] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7466.161350] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7473.220245] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7480.517473] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7480.903238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7481.917742] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7482.056954] LustreError: 241514:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7486.957546] LustreError: 241517:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7489.207055] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 7492.068402] LustreError: 241514:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7497.377190] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7502.307761] LustreError: 241589:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7502.346753] LustreError: 241589:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7504.999107] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7505.279360] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7510.515652] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7510.531804] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7511.647434] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 7521.579432] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7526.533941] Lustre: Permanently reactivating lustre-MDT0001 [ 7526.546415] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7526.550908] Lustre: Skipped 2 previous similar messages [ 7527.528160] Lustre: lustre-MDT0001: Client d7c8743f-776e-4db5-88dd-bb6fd8bbbfb9 (at 192.168.206.12@tcp) reconnecting [ 7532.572160] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7532.580726] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7532.586914] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7532.593943] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7534.755266] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7536.764593] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7543.834216] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7544.091270] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7549.220848] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7549.507501] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7552.463779] Lustre: Permanently deactivating lustre-MDT0001 [ 7552.500120] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7552.508666] Lustre: Skipped 2 previous similar messages [ 7561.346829] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7573.988427] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7573.990124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7574.014382] Lustre: Skipped 2 previous similar messages [ 7574.029829] Lustre: Skipped 3 previous similar messages [ 7577.726907] Lustre: server umount lustre-MDT0000 complete [ 7579.106097] LustreError: 241156:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7579.117096] LustreError: 241156:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7581.328202] LustreError: 240426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787582194 with bad export cookie 2968548914589558353 [ 7581.333267] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7581.340921] LustreError: 240426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7581.594736] Lustre: server umount lustre-MDT0001 complete [ 7587.827410] Lustre: server umount lustre-OST0000 complete [ 7601.313511] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 10:36:52 (1787582212) [ 7613.454791] Lustre: server umount lustre-OST0001 complete [ 7620.167882] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [ 7631.335856] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 7669.136625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7669.402491] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7669.419678] Lustre: Skipped 2 previous similar messages [ 7669.460194] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7669.572481] Lustre: lustre-MDT0000: new disk, initializing [ 7669.644723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7669.662799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7673.756101] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 7683.967101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7684.099974] Lustre: 247982:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7684.150034] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7684.154321] Lustre: Skipped 1 previous similar message [ 7684.214974] Lustre: lustre-MDT0001: new disk, initializing [ 7684.324052] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7684.379252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7684.413710] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7689.303362] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 7698.872387] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7704.510962] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7712.006656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7712.496778] Lustre: lustre-OST0000: new disk, initializing [ 7712.500786] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7712.512392] Lustre: 249951:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7712.688352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7719.552879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7719.571604] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7719.747605] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7720.409422] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 7731.070418] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7733.543843] LustreError: 247990:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7742.485251] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7742.776803] Lustre: lustre-OST0001: new disk, initializing [ 7742.781395] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7742.790336] Lustre: 251069:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7743.608764] LustreError: 247990:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7743.638206] LustreError: 250720:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7749.227700] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7749.239020] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7749.366257] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7749.710380] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 7753.706042] LustreError: 250720:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7762.435159] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7765.061524] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7770.091016] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7770.094135] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7770.095567] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7770.104734] LustreError: Skipped 1 previous similar message [ 7770.160494] Lustre: Skipped 2 previous similar messages [ 7775.204089] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7775.229419] Lustre: Skipped 2 previous similar messages [ 7780.339949] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7780.359021] Lustre: Skipped 1 previous similar message [ 7781.343293] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7781.451257] Lustre: server umount lustre-OST0001 complete [ 7785.458789] LustreError: 250261:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7785.510589] LustreError: 250261:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7800.801708] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7800.833565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7800.854916] Lustre: Skipped 1 previous similar message [ 7806.494560] Lustre: server umount lustre-OST0000 complete [ 7811.041317] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7811.066082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7811.094325] Lustre: Skipped 2 previous similar messages [ 7811.110324] Lustre: Skipped 3 previous similar messages [ 7817.375630] Lustre: server umount lustre-MDT0000 complete [ 7821.506830] LustreError: 247974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787582434 with bad export cookie 2968548914589560964 [ 7821.516279] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7821.525632] LustreError: 247974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7828.480088] Lustre: server umount lustre-MDT0001 complete [ 7843.098471] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 7847.704695] Key type lgssc unregistered [ 7848.179614] LNet: 253111:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7848.195636] LNetError: 253111:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7849.256590] LNet: Removed LNI 192.168.206.112@tcp [ 7850.474427] Key type .llcrypt unregistered [ 7850.477270] Key type ._llcrypt unregistered [ 7898.195666] Key type ._llcrypt registered [ 7898.198939] Key type .llcrypt registered [ 7898.300577] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 7899.385705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7899.429237] alg: No test for adler32 (adler32-zlib) [ 7901.534398] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [ 7902.155339] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 7903.999173] Key type lgssc registered [ 7906.044856] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7917.049563] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7917.068293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7918.352062] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7918.371715] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7918.548452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7923.077363] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 7932.788358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7932.854208] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7932.878288] Lustre: 255323:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7933.062501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7936.969780] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 7948.467309] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7956.420477] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7965.311193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7965.501051] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7965.825833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7972.121980] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7972.952438] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 7979.871732] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8033.769352] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8033.771718] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8033.775444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8033.780458] Lustre: Skipped 1 previous similar message [ 8033.803553] LustreError: Skipped 1 previous similar message [ 8038.836644] Lustre: server umount lustre-OST0000 complete [ 8043.276304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8057.929364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8058.347530] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8064.853909] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 8073.229892] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8073.717332] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8073.733808] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8073.741497] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8114.657680] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8114.677070] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8114.700835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8114.706208] Lustre: Skipped 1 previous similar message [ 8114.731421] Lustre: Skipped 2 previous similar messages [ 8119.849402] Lustre: server umount lustre-OST0000 complete [ 8127.456987] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8127.476315] LustreError: Skipped 1 previous similar message [ 8127.491753] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8127.527612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8130.017928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8130.025376] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8132.277164] Lustre: server umount lustre-MDT0000 complete [ 8137.228992] LustreError: 255314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787582750 with bad export cookie 4805254095651227417 [ 8137.237459] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8137.263511] LustreError: 255314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8137.749624] Lustre: server umount lustre-MDT0001 complete [ 8149.368452] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 8152.481775] Key type lgssc unregistered [ 8152.855728] LNet: 260748:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8152.877605] LNetError: 260748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8153.894915] LNet: Removed LNI 192.168.206.112@tcp [ 8154.991516] Key type .llcrypt unregistered [ 8155.000074] Key type ._llcrypt unregistered [ 8181.222060] Key type ._llcrypt registered [ 8181.225690] Key type .llcrypt registered [ 8181.352669] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 8182.231937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8182.247830] alg: No test for adler32 (adler32-zlib) [ 8183.740678] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [ 8184.056192] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 8185.800141] Key type lgssc registered [ 8187.310263] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8194.990309] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8195.010559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8196.487911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8201.187604] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 8206.820458] LustreError: 262167:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8206.859499] LustreError: 262167:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8209.920624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8210.411952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8215.606537] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 8223.500095] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8230.406256] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8239.451268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8240.175644] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8241.270541] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8248.451571] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 8257.565662] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8281.570555] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8281.572349] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8281.595191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8286.691910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8286.700962] Lustre: Skipped 2 previous similar messages [ 8287.460955] Lustre: server umount lustre-OST0000 complete [ 8291.809202] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8291.837464] Lustre: Skipped 1 previous similar message [ 8291.855092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8291.864030] Lustre: Skipped 1 previous similar message [ 8296.929395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8296.937811] Lustre: Skipped 1 previous similar message [ 8297.751522] Lustre: server umount lustre-MDT0000 complete [ 8302.050550] LustreError: 262898:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8302.079118] LustreError: 262898:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8302.327512] LustreError: 264013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787582915 with bad export cookie 11278775453343053201 [ 8302.337523] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8302.742646] Lustre: server umount lustre-MDT0001 complete [ 8317.229694] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 8320.745490] Key type lgssc unregistered [ 8321.026977] LNet: 266482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8321.033187] LNetError: 266482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8321.056921] LNet: Removed LNI 192.168.206.112@tcp [ 8321.930228] Key type .llcrypt unregistered [ 8321.931993] Key type ._llcrypt unregistered [ 8341.314100] Key type ._llcrypt registered [ 8341.320405] Key type .llcrypt registered [ 8341.490959] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 8342.689620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8342.743777] alg: No test for adler32 (adler32-zlib) [ 8344.000955] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [ 8344.326363] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 8346.023165] Key type lgssc registered [ 8347.161656] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8355.073050] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8355.083747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8356.467581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8361.213713] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 8366.560826] LustreError: 267702:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8366.587470] LustreError: 267702:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8370.168243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8370.643937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8375.480610] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 8383.449331] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8389.345779] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8396.383713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8396.772161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8401.918604] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8402.144720] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 8409.400182] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8431.590418] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8431.599025] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8431.604090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8431.604182] Lustre: Skipped 1 previous similar message [ 8435.399593] Lustre: server umount lustre-OST0000 complete [ 8441.825178] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8441.836514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8441.838768] Lustre: Skipped 1 previous similar message [ 8441.861924] Lustre: Skipped 2 previous similar messages [ 8445.382762] Lustre: server umount lustre-MDT0000 complete [ 8446.946565] LustreError: 267702:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8446.970618] LustreError: 267702:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8449.073762] LustreError: 267683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787583062 with bad export cookie 5523131798734402721 [ 8449.075420] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8449.100866] LustreError: 267683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8449.601816] Lustre: server umount lustre-MDT0001 complete [ 8461.295093] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 8464.067322] Key type lgssc unregistered [ 8464.421156] LNet: 271406:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8464.441489] LNetError: 271406:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8464.462476] LNet: Removed LNI 192.168.206.112@tcp [ 8465.602605] Key type .llcrypt unregistered [ 8465.605070] Key type ._llcrypt unregistered [ 8494.518301] Key type ._llcrypt registered [ 8494.520654] Key type .llcrypt registered [ 8494.623238] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 8496.173072] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8496.214143] alg: No test for adler32 (adler32-zlib) [ 8497.431389] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [ 8497.762080] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 8499.479164] Key type lgssc registered [ 8500.808172] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8509.042631] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8509.081068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8510.591282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8515.460512] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 8520.163861] LustreError: 272824:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8520.177361] LustreError: 272824:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8523.579946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8523.941950] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8528.718380] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 8536.108493] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8541.497450] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8547.760567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8548.059594] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8554.209974] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 8555.267385] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8561.548888] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8590.305894] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8590.319264] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8590.334969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8591.334548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8591.336643] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8591.337398] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8591.340758] Lustre: Skipped 1 previous similar message [ 8594.421244] Lustre: server umount lustre-OST0000 complete [ 8600.547341] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8600.560531] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8600.582246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8606.688609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8606.692820] Lustre: Skipped 3 previous similar messages [ 8611.811037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8611.824614] Lustre: Skipped 1 previous similar message [ 8612.319752] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8612.634172] Lustre: server umount lustre-MDT0000 complete [ 8615.904850] LustreError: 273552:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8615.914243] LustreError: 273552:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8616.600097] LustreError: 276380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787583229 with bad export cookie 6454782075589085514 [ 8616.602114] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8616.611022] LustreError: 276380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8616.973295] Lustre: server umount lustre-MDT0001 complete [ 8628.807567] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 8631.581440] Key type lgssc unregistered [ 8631.950656] LNet: 277136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8631.959491] LNetError: 277136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8631.979417] LNet: Removed LNI 192.168.206.112@tcp [ 8632.915996] Key type .llcrypt unregistered [ 8632.918919] Key type ._llcrypt unregistered [ 8651.676729] Key type ._llcrypt registered [ 8651.679609] Key type .llcrypt registered [ 8651.807839] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 8652.898125] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8653.123076] alg: No test for adler32 (adler32-zlib) [ 8654.124523] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [ 8654.335167] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 8656.007154] Key type lgssc registered [ 8656.913755] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8663.729318] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8663.741971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8665.254176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8669.876872] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 8675.821035] LustreError: 278374:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8675.850182] LustreError: 278374:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8678.113565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8678.455264] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8683.087035] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 8690.632387] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8696.767970] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8703.727376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8703.918496] LustreError: 280218:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8704.127557] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8710.667402] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 8710.668098] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8719.477646] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8735.207953] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8735.210227] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8735.247633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8736.227172] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8736.239365] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8736.258953] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8737.165978] Lustre: server umount lustre-OST0000 complete [ 8741.346481] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8741.351553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8741.365349] Lustre: Skipped 1 previous similar message [ 8741.385547] Lustre: Skipped 2 previous similar messages [ 8746.467660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8746.481041] Lustre: Skipped 1 previous similar message [ 8751.587293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8751.604426] Lustre: Skipped 1 previous similar message [ 8755.167552] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8755.301633] Lustre: server umount lustre-MDT0000 complete [ 8756.704334] LustreError: 278374:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8756.732252] LustreError: 278374:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8759.861113] LustreError: 278341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787583372 with bad export cookie 12507232598885931229 [ 8759.870678] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8759.883222] LustreError: 278341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8760.311198] Lustre: server umount lustre-MDT0001 complete [ 8771.552213] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 8774.694923] Key type lgssc unregistered [ 8775.010746] LNet: 282077:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8775.043434] LNetError: 282077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8776.101852] LNet: Removed LNI 192.168.206.112@tcp [ 8777.080439] Key type .llcrypt unregistered [ 8777.083332] Key type ._llcrypt unregistered [ 8789.665800] Key type ._llcrypt registered [ 8789.672196] Key type .llcrypt registered [ 8789.830992] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8807.255656] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [ 8822.417796] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 8823.318132] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8823.381657] alg: No test for adler32 (adler32-zlib) [ 8824.537573] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [ 8824.834364] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 8826.511174] Key type lgssc registered [ 8827.873922] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8865.296394] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8865.329582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8866.610893] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8866.651900] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8866.744474] Lustre: lustre-MDT0000: new disk, initializing [ 8866.813200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8866.829770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8872.295222] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 8884.258735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8884.377740] Lustre: 285630:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8884.431584] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8884.435863] Lustre: Skipped 1 previous similar message [ 8884.500120] Lustre: lustre-MDT0001: new disk, initializing [ 8884.579474] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8884.623874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8884.641188] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8888.979549] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 8898.782901] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8904.658641] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8912.479484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8912.832490] Lustre: lustre-OST0000: new disk, initializing [ 8912.840117] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8912.855048] Lustre: 287599:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8912.965992] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8920.746648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8920.764794] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8920.961880] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8921.440461] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 8933.672259] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8939.277950] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8939.478834] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8943.707268] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8943.911631] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8946.669553] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8946.676511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8946.683452] Lustre: Skipped 1 previous similar message [ 8946.695323] Lustre: Skipped 1 previous similar message [ 8951.842079] Lustre: server umount lustre-OST0000 complete [ 8956.385090] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8956.395790] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8956.427518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8961.766074] Lustre: server umount lustre-MDT0000 complete [ 8965.359154] LustreError: 285623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787583578 with bad export cookie 12240185291659200622 [ 8965.360044] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8965.373023] LustreError: 285623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8965.799611] Lustre: server umount lustre-MDT0001 complete [ 8974.121743] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 10:59:46 (1787583586) [ 8980.742892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8981.117653] LustreError: 290012:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8981.143256] LustreError: 290012:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8981.193217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8986.255794] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 8986.598327] LustreError: 290013:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8991.719376] LustreError: 290012:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8995.476254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8995.833616] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9000.097156] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 9007.687566] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9013.889877] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9020.296208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9020.645676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9026.709554] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 9034.030932] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9050.595282] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9050.598213] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9050.607937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9050.615767] Lustre: Skipped 2 previous similar messages [ 9050.627192] Lustre: Skipped 1 previous similar message [ 9054.475270] Lustre: server umount lustre-OST0000 complete [ 9060.833618] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9060.850911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9060.864389] Lustre: Skipped 2 previous similar messages [ 9064.717333] Lustre: server umount lustre-MDT0000 complete [ 9065.952295] LustreError: 290520:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9065.973064] LustreError: 290520:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9068.371898] LustreError: 289994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787583681 with bad export cookie 12240185291659201504 [ 9068.379298] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9068.384063] LustreError: 289994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9068.743310] Lustre: server umount lustre-MDT0001 complete [ 9080.949369] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 9083.936468] Key type lgssc unregistered [ 9084.299569] LNet: 293584:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9084.306478] LNetError: 293584:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9084.325179] LNet: Removed LNI 192.168.206.112@tcp [ 9085.179678] Key type .llcrypt unregistered [ 9085.183234] Key type ._llcrypt unregistered [ 9111.025327] Key type ._llcrypt registered [ 9111.028455] Key type .llcrypt registered [ 9111.121362] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [ 9126.630559] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 9128.007932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9128.066629] alg: No test for adler32 (adler32-zlib) [ 9129.203650] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [ 9129.413238] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 9131.055306] Key type lgssc registered [ 9132.345544] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9179.997641] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9180.018299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9181.216797] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9181.254671] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9181.319153] Lustre: lustre-MDT0000: new disk, initializing [ 9181.408050] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9181.422568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9186.192499] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 9197.194615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9197.330629] Lustre: 297413:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9197.393275] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9197.397877] Lustre: Skipped 1 previous similar message [ 9197.523986] Lustre: lustre-MDT0001: new disk, initializing [ 9197.624657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9197.668711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9197.682080] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9202.991811] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 9214.671901] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9221.114313] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9228.918995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9229.284492] Lustre: lustre-OST2710: new disk, initializing [ 9229.289914] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9229.298572] Lustre: 299383:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9229.422546] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9235.004160] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9235.015849] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9235.140400] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9235.610462] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 9246.789200] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9254.034570] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9254.334596] Lustre: lustre-OST03e8: new disk, initializing [ 9254.337929] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9254.344922] Lustre: 300446:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9254.449821] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9261.514042] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 9261.700926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9261.710819] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9261.907128] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9270.313379] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9278.183588] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9278.389137] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9283.106184] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9283.294978] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9287.761976] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9287.965595] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9292.438434] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9292.638736] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9297.893160] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9297.896374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9297.904421] Lustre: Skipped 3 previous similar messages [ 9297.915498] Lustre: Skipped 3 previous similar messages [ 9299.315670] Lustre: server umount lustre-MDT0000 complete [ 9302.937631] LustreError: 299382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787583916 with bad export cookie 8630569927102079872 [ 9302.944459] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9302.954081] LustreError: 299382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9303.020358] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9303.030892] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9303.038160] Lustre: Skipped 1 previous similar message [ 9303.416309] Lustre: server umount lustre-MDT0001 complete [ 9309.476629] Lustre: server umount lustre-OST2710 complete [ 9313.061899] Lustre: server umount lustre-OST03e8 complete [ 9329.820885] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [ 9339.776760] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 9372.818606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9372.996566] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9373.020552] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9373.090301] Lustre: lustre-MDT0000: new disk, initializing [ 9373.148437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9373.163418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9377.042851] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 9386.904304] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9386.998790] Lustre: 305697:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9387.081540] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9387.088114] Lustre: Skipped 1 previous similar message [ 9387.189734] Lustre: lustre-MDT0001: new disk, initializing [ 9387.300360] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9387.366698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9387.383338] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9391.104943] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 9400.337049] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9405.837144] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9413.119134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9413.487430] Lustre: lustre-OST0000: new disk, initializing [ 9413.492843] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9413.498872] Lustre: 307662:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9413.567973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9419.188312] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 9421.913763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9421.930916] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9422.040443] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9426.881405] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9431.706888] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9431.923043] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9436.255309] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9436.410620] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9442.786938] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9442.787918] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9442.789954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9442.789960] Lustre: Skipped 1 previous similar message [ 9442.796263] LustreError: Skipped 1 previous similar message [ 9442.802406] Lustre: Skipped 1 previous similar message [ 9444.291649] Lustre: server umount lustre-OST0000 complete [ 9447.905242] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9447.909406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9447.918136] Lustre: Skipped 1 previous similar message [ 9447.927557] Lustre: Skipped 2 previous similar messages [ 9453.025480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9453.036317] Lustre: Skipped 1 previous similar message [ 9453.957995] Lustre: server umount lustre-MDT0000 complete [ 9456.812216] LustreError: 305687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787584069 with bad export cookie 8630569927102081545 [ 9456.817880] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9456.820455] LustreError: 305687:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9457.015638] Lustre: server umount lustre-MDT0001 complete [ 9464.293453] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 11:07:56 (1787584076) [ 9465.714477] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9467.065528] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 11:07:59 (1787584079) [ 9469.976137] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 9497.032236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9497.399666] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9497.417990] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9497.519978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9503.133515] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 9513.386448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9513.505475] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9513.558571] Lustre: 311516:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9518.675589] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 9541.738667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9541.980625] Lustre: Denying initial registration attempt from nid 192.168.206.112@tcp, specified as failover [ 9541.987278] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9541.998685] LustreError: 312972:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -99 [ 9542.003753] LustreError: 312972:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9542.011753] LustreError: 312972:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9542.096945] Lustre: server umount lustre-OST0000 complete [ 9542.100911] LustreError: 312972:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9549.794296] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9549.807678] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9549.829781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9554.086302] Lustre: server umount lustre-MDT0000 complete [ 9557.029390] LustreError: 311508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787584170 with bad export cookie 8630569927102082434 [ 9557.035083] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9557.040258] LustreError: 311508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9557.292362] Lustre: server umount lustre-MDT0001 complete [ 9565.967918] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 9568.185633] Key type lgssc unregistered [ 9568.437869] LNet: 314096:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9568.444091] LNetError: 314096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9568.454211] LNet: Removed LNI 192.168.206.112@tcp [ 9569.091216] Key type .llcrypt unregistered [ 9569.094866] Key type ._llcrypt unregistered [ 9580.600345] Key type ._llcrypt registered [ 9580.607693] Key type .llcrypt registered [ 9580.751985] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 9581.594586] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9581.622876] alg: No test for adler32 (adler32-zlib) [ 9582.741604] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [ 9583.104591] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 9584.839222] Key type lgssc registered [ 9586.122474] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9634.162593] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 9643.116094] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9643.129974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9644.471473] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9644.501531] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9644.730220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9649.533201] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 9660.282777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9660.372612] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9660.425053] Lustre: 316863:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9660.807658] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9664.874353] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 9672.466630] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9680.005934] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9703.642949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9704.270859] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9704.789732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9716.485782] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 9724.647677] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9726.947883] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9726.954048] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9726.964694] Lustre: Skipped 1 previous similar message [ 9726.966755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9726.984795] Lustre: Skipped 1 previous similar message [ 9732.067206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9732.072882] Lustre: Skipped 1 previous similar message [ 9737.195219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9737.217965] Lustre: Skipped 1 previous similar message [ 9741.280308] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9741.524626] Lustre: server umount lustre-OST0000 complete [ 9747.425619] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9747.445728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9747.453655] Lustre: Skipped 1 previous similar message [ 9752.302563] Lustre: server umount lustre-MDT0000 complete [ 9756.295525] LustreError: 318730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787584369 with bad export cookie 5673969843468246865 [ 9756.310590] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9756.313481] LustreError: 318730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9756.826198] Lustre: server umount lustre-MDT0001 complete [ 9767.863779] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 9770.734265] Key type lgssc unregistered [ 9771.026350] LNet: 320444:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9771.036874] LNetError: 320444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9771.057921] LNet: Removed LNI 192.168.206.112@tcp [ 9771.819690] Key type .llcrypt unregistered [ 9771.821799] Key type ._llcrypt unregistered [ 9796.347345] Key type ._llcrypt registered [ 9796.350247] Key type .llcrypt registered [ 9796.433708] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [ 9797.590834] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9797.654168] alg: No test for adler32 (adler32-zlib) [ 9798.687761] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [ 9798.959357] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [ 9800.663805] Key type lgssc registered [ 9802.079384] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9810.076278] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9810.097627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9811.559529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9816.686793] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 9821.176462] LustreError: 321861:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9821.195294] LustreError: 321861:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9825.538339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9825.924992] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9830.124563] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 9838.266948] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9844.371253] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9853.944889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9854.286803] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9860.727761] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 9869.165657] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9881.073429] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9881.078066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9881.086926] Lustre: Skipped 2 previous similar messages [ 9881.102375] Lustre: Skipped 2 previous similar messages [ 9881.588749] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.12@tcp (stopping) [ 9884.164291] Lustre: server umount lustre-MDT0000 complete [ 9886.181547] LustreError: 321861:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9886.207507] LustreError: 321861:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9888.001866] LustreError: 322594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787584501 with bad export cookie 7784249881115701331 [ 9888.005202] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9888.011459] LustreError: 322594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9891.296840] LustreError: 321856:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9891.297954] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9891.328141] LustreError: 321856:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9891.356339] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9893.344876] LustreError: 324444:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9893.360943] LustreError: 324444:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9894.502849] Lustre: server umount lustre-MDT0001 complete [ 9901.497697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9910.232543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9913.838592] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x6c0736062ec74c3e [ 9913.855419] Lustre: MGC192.168.206.112@tcp: Connection restored to 0@lo (at 0@lo) [ 9914.173222] LustreError: 323694:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9914.259342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9914.342334] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9918.577702] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 9923.497265] LustreError: 325454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9923.510364] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9923.526621] LustreError: 325454:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9927.348449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9927.769242] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9932.592146] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [ 9932.789845] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9939.948236] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9941.403370] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9947.344043] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9949.289880] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9955.296376] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9955.308389] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9955.327769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9955.332598] Lustre: Skipped 2 previous similar messages [ 9958.368694] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9958.389097] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9960.449305] Lustre: server umount lustre-OST0000 complete [ 9968.613811] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9968.619448] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9968.635962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9968.656030] Lustre: Skipped 3 previous similar messages [ 9970.537826] Lustre: server umount lustre-MDT0000 complete [ 9973.728409] LustreError: 325453:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9973.746126] LustreError: 325453:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9974.090970] LustreError: 322594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787584587 with bad export cookie 7784249881115708478 [ 9974.100517] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9974.105613] LustreError: 322594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9974.444776] Lustre: server umount lustre-MDT0001 complete [ 9986.407753] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [ 9989.608829] Key type lgssc unregistered [ 9989.903693] LNet: 328132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9989.916510] LNetError: 328132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9989.954767] LNet: Removed LNI 192.168.206.112@tcp [ 9990.748650] Key type .llcrypt unregistered [ 9990.755597] Key type ._llcrypt unregistered [10014.553934] Key type ._llcrypt registered [10014.555716] Key type .llcrypt registered [10014.749516] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [10015.701792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10015.856183] alg: No test for adler32 (adler32-zlib) [10017.023984] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [10017.305639] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [10019.023823] Key type lgssc registered [10020.007653] Lustre: Echo OBD driver; http://www.lustre.org/ [10027.641722] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10027.671672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10029.048031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10032.711754] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [10039.266978] LustreError: 329549:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10039.283254] LustreError: 329549:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10040.203571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10040.603702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10044.778580] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [10052.523407] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10058.527197] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10064.353769] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10064.358054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10064.366427] Lustre: Skipped 1 previous similar message [10064.374237] Lustre: Skipped 1 previous similar message [10065.378194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10065.387216] Lustre: Skipped 1 previous similar message [10070.497341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10070.508941] Lustre: Skipped 1 previous similar message [10075.616753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10075.627130] Lustre: Skipped 1 previous similar message [10076.127131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10076.343646] Lustre: server umount lustre-MDT0000 complete [10080.452289] LustreError: 329530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787584693 with bad export cookie 13602226205172537073 [10080.455637] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10080.464434] LustreError: 329530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10080.942543] Lustre: server umount lustre-MDT0001 complete [10096.351795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10096.624464] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10096.663509] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10096.884254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10101.399636] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [10109.821892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10109.934774] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10109.964730] Lustre: 332141:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10110.216073] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10114.307679] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [10122.445976] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10129.082590] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10139.881293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10140.187284] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10140.200705] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10140.213916] LustreError: 334023:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [10140.236411] LustreError: 334023:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [10140.253749] LustreError: 334023:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10140.360772] Lustre: server umount lustre-OST0000 complete [10140.366077] LustreError: 334023:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10151.211205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10151.441185] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10151.641357] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10157.962586] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [10159.748041] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10165.859403] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10173.357987] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10173.420153] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10173.487991] Lustre: lustre-OST0001: new disk, initializing [10173.491645] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10173.499741] Lustre: 335566:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10173.575683] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10179.348475] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [10182.252101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10182.267424] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10182.407356] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10186.846890] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10192.864156] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10192.869110] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10192.883720] LustreError: Skipped 1 previous similar message [10192.900441] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10192.920680] Lustre: Skipped 1 previous similar message [10203.113269] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10203.116682] Lustre: Skipped 3 previous similar messages [10204.127338] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10204.331883] Lustre: server umount lustre-OST0001 complete [10208.237050] LustreError: 334489:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10208.238725] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10208.254793] LustreError: 334489:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10208.285078] Lustre: Skipped 1 previous similar message [10213.356094] LustreError: 334488:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10213.369790] LustreError: 334488:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10214.438405] Lustre: server umount lustre-OST0000 complete [10218.465796] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10218.482822] Lustre: Skipped 2 previous similar messages [10223.584943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10223.595503] Lustre: Skipped 7 previous similar messages [10232.287583] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10233.824872] LustreError: 332152:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10233.847211] LustreError: 332152:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10234.515642] Lustre: server umount lustre-MDT0000 complete [10237.612128] LustreError: 332132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787584850 with bad export cookie 13602226205172537626 [10237.624591] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10237.625307] LustreError: 332132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10238.023761] Lustre: server umount lustre-MDT0001 complete [10247.997154] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [10251.799500] Key type lgssc unregistered [10252.144140] LNet: 337526:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10252.179070] LNetError: 337526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10252.201161] LNet: Removed LNI 192.168.206.112@tcp [10253.209074] Key type .llcrypt unregistered [10253.210944] Key type ._llcrypt unregistered [10326.368877] Key type ._llcrypt registered [10326.371707] Key type .llcrypt registered [10326.467966] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [10341.061311] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [10342.098092] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10342.301981] alg: No test for adler32 (adler32-zlib) [10343.372719] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [10343.624178] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [10345.312898] Key type lgssc registered [10346.659543] Lustre: Echo OBD driver; http://www.lustre.org/ [10381.175957] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10381.266500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10382.715291] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10382.773629] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10382.880757] Lustre: lustre-MDT0000: new disk, initializing [10383.036246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10383.073467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10387.813558] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [10398.447811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10398.517393] Lustre: 342464:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10398.566783] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10398.570413] Lustre: Skipped 1 previous similar message [10398.628534] Lustre: lustre-MDT0001: new disk, initializing [10398.702824] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10398.754450] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10398.765492] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10403.147745] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [10412.983362] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10418.875354] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10427.010330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10427.393751] Lustre: lustre-OST0000: new disk, initializing [10427.404617] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10427.417793] Lustre: 344434:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10427.551653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10433.656306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10433.677156] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10433.799708] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10434.428675] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [10444.608982] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10449.703746] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10449.903430] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10455.599316] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10455.845617] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10459.627075] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10459.629703] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10459.635740] LustreError: Skipped 1 previous similar message [10459.639937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10459.657716] Lustre: Skipped 1 previous similar message [10464.014879] Lustre: server umount lustre-OST0000 complete [10469.857295] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10469.859909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10469.878205] Lustre: Skipped 1 previous similar message [10469.882019] Lustre: Skipped 1 previous similar message [10474.983831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10474.999204] Lustre: Skipped 2 previous similar messages [10479.075722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10479.094975] Lustre: Skipped 1 previous similar message [10482.144681] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10482.492214] Lustre: server umount lustre-MDT0000 complete [10486.094941] LustreError: 342457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787585099 with bad export cookie 3347935199285522191 [10486.104385] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10486.115937] LustreError: 342457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10486.496981] Lustre: server umount lustre-MDT0001 complete [10495.658876] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 11:25:07 (1787585107) [10503.763080] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 11:25:15 (1787585115) [10514.103580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10514.530651] LustreError: 347189:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10514.545864] LustreError: 347189:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10514.595662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10518.981867] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [10520.033160] LustreError: 347190:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10524.128481] LustreError: 347189:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10527.022368] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10527.388864] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10531.661233] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [10539.695215] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10545.846550] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10553.968696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10554.533931] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10561.348521] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [10569.479614] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10577.558806] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10579.277339] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10580.768313] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10582.420178] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10584.206966] Lustre: Failing over lustre-MDT0000 [10584.518392] Lustre: server umount lustre-MDT0000 complete [10587.619166] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10587.625828] LustreError: 347185:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10587.640420] Lustre: Skipped 1 previous similar message [10587.665856] LustreError: 347185:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10592.715489] LustreError: 348623:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10602.955192] LustreError: 348623:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10602.975109] Lustre: 340687:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787585200/real 1787585200] req@00000000898e8255 x1874418682894720/t0(0) o400->MGC192.168.206.112@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787585216 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10602.975776] LustreError: 348623:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10603.043341] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10603.084027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10613.216151] LustreError: 340685:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005417e5dc x1874418682900096/t0(0) o250->MGC192.168.206.112@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10613.491401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10613.552406] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10617.859508] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [10618.306664] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10618.875169] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10618.927187] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10618.968662] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10627.803802] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10629.366218] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10632.833298] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10634.613787] Lustre: Failing over lustre-MDT0000 [10634.830894] Lustre: server umount lustre-MDT0000 complete [10638.818975] LustreError: 349868:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10638.847880] LustreError: 349868:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [10639.328750] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10639.334645] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10639.352731] Lustre: Skipped 2 previous similar messages [10652.613806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10652.709840] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10652.957928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10653.005629] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10654.143645] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10657.123781] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [10658.290902] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10658.305600] Lustre: Skipped 2 previous similar messages [10658.342376] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [10658.392530] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10665.586493] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10667.180655] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10673.632472] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10673.644163] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10673.644741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10673.646481] Lustre: Skipped 2 previous similar messages [10679.333723] Lustre: server umount lustre-MDT0000 complete [10682.890977] LustreError: 349032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787585296 with bad export cookie 3347935199285524900 [10682.896779] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10682.905434] LustreError: 349032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10683.228291] Lustre: server umount lustre-MDT0001 complete [10697.346429] Lustre: server umount lustre-OST0000 complete [10712.071728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10712.520725] LustreError: 353530:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10712.546433] LustreError: 353530:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [10716.958469] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [10724.350134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10724.738377] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10724.743684] Lustre: Skipped 1 previous similar message [10728.873751] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [10735.560810] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10740.820356] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10746.620857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10751.828899] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [10757.236584] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10758.386419] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10762.722682] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10762.734754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10762.748551] Lustre: Skipped 2 previous similar messages [10762.752522] Lustre: Skipped 5 previous similar messages [10762.761192] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10764.388867] Lustre: server umount lustre-OST0000 complete [10770.913586] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10774.712756] Lustre: server umount lustre-MDT0000 complete [10778.089154] LustreError: 354585:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10778.105659] LustreError: 354585:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10780.134488] LustreError: 353510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787585393 with bad export cookie 3347935199285525649 [10780.143223] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10780.165369] LustreError: 353510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10780.777454] Lustre: server umount lustre-MDT0001 complete [10796.187441] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [10798.692829] Key type lgssc unregistered [10799.004740] LNet: 357068:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10799.012181] LNetError: 357068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10799.036211] LNet: Removed LNI 192.168.206.112@tcp [10800.147958] Key type .llcrypt unregistered [10800.150164] Key type ._llcrypt unregistered [10822.726673] Key type ._llcrypt registered [10822.728502] Key type .llcrypt registered [10822.808739] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [10823.718383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10823.744430] alg: No test for adler32 (adler32-zlib) [10824.797418] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [10825.042102] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [10826.759186] Key type lgssc registered [10827.816924] Lustre: Echo OBD driver; http://www.lustre.org/ [10834.439042] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10834.477466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10835.977946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10839.781274] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [10846.180551] LustreError: 358564:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10846.195698] LustreError: 358564:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10848.126283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10848.544864] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10852.971435] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [10860.400082] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10864.858531] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10870.450619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10870.721640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10876.037204] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [10881.015230] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10883.230242] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10894.592221] Lustre: server umount lustre-MDT0000 complete [10894.819730] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10894.830588] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10894.857113] LustreError: 358564:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10896.361579] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10896.362683] LustreError: 359593:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10896.393715] Lustre: Skipped 1 previous similar message [10896.404664] LustreError: 359593:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10898.246858] LustreError: 358545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787585511 with bad export cookie 5826608725728122891 [10898.257824] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10898.259415] LustreError: 358545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10898.713822] Lustre: server umount lustre-MDT0001 complete [10914.070245] Lustre: server umount lustre-OST0000 complete [10935.037313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10935.838120] LustreError: 362274:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10935.929188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10940.602570] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [10941.417333] LustreError: 362275:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10951.644398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10951.654922] LustreError: 362275:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10951.691369] LustreError: 362275:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10951.965907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10957.725741] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [10966.802486] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10972.110615] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10978.123581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10978.644229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10981.712167] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10985.263110] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [10995.148495] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11011.562487] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11011.587592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11011.592475] Lustre: Skipped 1 previous similar message [11015.386655] Lustre: server umount lustre-MDT0000 complete [11016.687651] LustreError: 362270:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11016.715581] LustreError: 362270:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11019.739951] LustreError: 363009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787585632 with bad export cookie 5826608725728124004 [11019.746731] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11019.760774] LustreError: 363009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11020.259629] Lustre: server umount lustre-MDT0001 complete [11035.508894] Lustre: server umount lustre-OST0000 complete [11050.298508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11050.843086] LustreError: 366188:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11050.922096] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11055.585437] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [11064.118738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11068.900116] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [11075.719236] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11081.415797] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11088.250169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11088.610687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11088.615202] Lustre: Skipped 1 previous similar message [11093.971490] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [11095.789379] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11100.679724] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11106.275426] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11106.275504] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11106.284689] Lustre: Skipped 2 previous similar messages [11106.305205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11106.312249] Lustre: Skipped 1 previous similar message [11111.398975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11111.412514] Lustre: Skipped 1 previous similar message [11111.711379] Lustre: server umount lustre-OST0000 complete [11116.014510] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11116.025152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11116.051368] Lustre: Skipped 1 previous similar message [11116.062156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11121.632265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11121.641608] Lustre: Skipped 1 previous similar message [11130.337140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11130.463842] Lustre: server umount lustre-MDT0000 complete [11131.880227] LustreError: 366916:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11131.912391] LustreError: 366916:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11134.465400] LustreError: 366170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787585747 with bad export cookie 5826608725728125026 [11134.468203] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11134.472373] LustreError: 366170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11134.813216] Lustre: server umount lustre-MDT0001 complete [11145.344695] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [11149.044942] Key type lgssc unregistered [11149.415556] LNet: 369723:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11149.425718] LNetError: 369723:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11149.441510] LNet: Removed LNI 192.168.206.112@tcp [11150.302415] Key type .llcrypt unregistered [11150.306627] Key type ._llcrypt unregistered [11171.787028] Key type ._llcrypt registered [11171.790626] Key type .llcrypt registered [11171.898770] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [11172.736730] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11172.880671] alg: No test for adler32 (adler32-zlib) [11173.834402] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [11174.094244] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [11175.801465] Key type lgssc registered [11176.877372] Lustre: Echo OBD driver; http://www.lustre.org/ [11183.484170] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11183.497148] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11183.503430] LustreError: 371143:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11183.521841] LustreError: 371143:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11183.537448] LustreError: 371143:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11183.546621] LustreError: 371143:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11183.552706] LustreError: 371143:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11189.855268] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11189.862455] LustreError: 371540:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11189.877486] LustreError: 371540:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11189.884055] LustreError: 371540:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11189.891644] LustreError: 371540:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11189.903304] LustreError: 371540:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11204.806908] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [11207.082535] Key type lgssc unregistered [11207.360491] LNet: 372546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11207.375092] LNetError: 372546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11207.393078] LNet: Removed LNI 192.168.206.112@tcp [11208.073148] Key type .llcrypt unregistered [11208.075694] Key type ._llcrypt unregistered [11222.881082] Key type ._llcrypt registered [11222.883257] Key type .llcrypt registered [11222.997111] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [11238.133934] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [11239.339033] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11239.451119] alg: No test for adler32 (adler32-zlib) [11240.587442] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [11240.854562] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [11242.551181] Key type lgssc registered [11244.218568] Lustre: Echo OBD driver; http://www.lustre.org/ [11299.323440] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11299.353201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11300.727898] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11300.787593] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11300.905300] Lustre: lustre-MDT0000: new disk, initializing [11301.044036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11301.074547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11306.192667] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [11317.848466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11318.063877] Lustre: 375703:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11318.150739] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11318.153129] Lustre: Skipped 1 previous similar message [11318.255510] Lustre: lustre-MDT0001: new disk, initializing [11318.365991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11318.464265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11318.469660] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11323.722942] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [11333.921582] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11340.559184] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11348.235501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11348.733539] Lustre: lustre-OST0000: new disk, initializing [11348.740976] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11348.754903] Lustre: 377676:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11348.940943] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11357.450097] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [11357.833059] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11357.849934] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11357.967057] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11368.088905] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11373.022973] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11373.257388] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11377.459958] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11377.700220] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11383.789317] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11383.790054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11383.808961] Lustre: Skipped 1 previous similar message [11383.820869] Lustre: Skipped 1 previous similar message [11386.064724] Lustre: server umount lustre-OST0000 complete [11394.027791] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11394.050240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11394.065053] Lustre: Skipped 1 previous similar message [11394.087143] Lustre: Skipped 1 previous similar message [11397.152910] Lustre: server umount lustre-MDT0000 complete [11402.146398] LustreError: 375695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787586015 with bad export cookie 2186991380322119389 [11402.150631] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11402.171683] LustreError: 375695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11402.842425] Lustre: server umount lustre-MDT0001 complete [11414.592380] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11416.404111] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 11:40:28 (1787586028) [11423.090782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11423.549517] LustreError: 380138:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11423.567599] LustreError: 380138:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11423.620618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11428.259468] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [11428.843918] LustreError: 380139:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11432.931555] LustreError: 380138:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11435.715797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11436.105947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11440.285397] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [11447.428899] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11453.363295] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11459.539304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11459.835173] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11465.445423] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [11472.851675] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11479.618766] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11479.738993] Lustre: lustre-OST0001: new disk, initializing [11479.742253] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11479.750980] Lustre: 382975:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11479.842564] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11485.329259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11485.340286] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11485.458188] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11486.182501] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [11496.511879] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11501.037511] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11501.037680] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11501.047384] LustreError: Skipped 1 previous similar message [11501.049663] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11501.064403] Lustre: Skipped 1 previous similar message [11503.586070] Lustre: lustre-OST0001: Not available for connect from 192.168.206.12@tcp (stopping) [11503.596014] Lustre: Skipped 1 previous similar message [11506.499331] Lustre: server umount lustre-OST0001 complete [11508.697631] LustreError: 383279:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11516.390717] LustreError: 381958:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11516.410148] LustreError: 381958:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11524.583203] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11524.598908] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11524.656286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11524.672994] Lustre: Skipped 2 previous similar messages [11526.635620] LustreError: 381959:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11526.653051] LustreError: 381959:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11536.867677] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11536.888210] Lustre: Skipped 5 previous similar messages [11538.399132] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11538.632817] Lustre: server umount lustre-OST0000 complete [11543.522520] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11543.528829] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11543.538790] Lustre: Skipped 1 previous similar message [11548.876877] Lustre: server umount lustre-MDT0000 complete [11552.234591] LustreError: 380139:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11552.267410] LustreError: 380139:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11553.576902] LustreError: 380119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787586166 with bad export cookie 2186991380322120271 [11553.580286] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11553.603074] LustreError: 380119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11560.177163] Lustre: server umount lustre-MDT0001 complete [11570.927612] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [11574.231098] Key type lgssc unregistered [11574.540501] LNet: 384965:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11574.549355] LNetError: 384965:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11574.562767] LNet: Removed LNI 192.168.206.112@tcp [11575.373682] Key type .llcrypt unregistered [11575.381456] Key type ._llcrypt unregistered [11630.530370] Key type ._llcrypt registered [11630.539363] Key type .llcrypt registered [11630.686994] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [11632.498208] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11632.634446] alg: No test for adler32 (adler32-zlib) [11633.764205] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [11634.021505] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [11635.826088] Key type lgssc registered [11637.408191] Lustre: Echo OBD driver; http://www.lustre.org/ [11646.062302] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11646.095339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11647.410168] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11647.440602] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11647.615541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11652.662710] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [11660.973270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11661.069061] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11661.117704] Lustre: 387767:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11661.450962] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11665.764839] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [11673.586233] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11679.402637] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11681.762872] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11681.775362] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11681.788382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11682.784981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11682.787922] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11688.047217] Lustre: server umount lustre-MDT0000 complete [11691.666655] LustreError: 387758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787586304 with bad export cookie 2863782722579274112 [11691.679131] LustreError: 387758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11691.679205] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11692.014835] Lustre: server umount lustre-MDT0001 complete [11703.721314] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 11:45:15 (1787586315) [11710.190610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11710.625601] LustreError: 390280:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11710.633135] LustreError: 390280:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11710.680844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11715.699646] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [11716.069278] LustreError: 390281:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11721.195596] LustreError: 390280:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11725.294567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11725.649613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11730.363468] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [11738.728555] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11744.722830] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11751.548052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11751.876101] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11752.101837] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11759.043118] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [11766.818349] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11774.874917] Lustre: Permanently deactivating lustre-OST0000 [11774.893806] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11782.206381] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11783.355722] LustreError: 393266:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11783.363434] LustreError: 393266:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11783.369746] LustreError: 393266:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11785.483698] LustreError: 393314:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11785.489344] LustreError: 393314:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11793.295981] Lustre: server umount lustre-OST0000 complete [11796.919476] LustreError: 393564:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11796.931869] LustreError: 393564:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11801.569505] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11801.573607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11801.601582] Lustre: Skipped 1 previous similar message [11801.612805] Lustre: Skipped 2 previous similar messages [11806.688830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11806.702855] Lustre: Skipped 1 previous similar message [11809.920691] Lustre: server umount lustre-MDT0000 complete [11811.809486] LustreError: 390280:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11811.844059] LustreError: 390280:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11814.452816] LustreError: 392125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787586427 with bad export cookie 2863782722579274693 [11814.460259] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11814.475756] LustreError: 392125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11824.285746] Lustre: server umount lustre-MDT0001 complete [11833.360517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11837.552770] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [11840.188872] LustreError: 394724:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11845.950250] Lustre: 395063:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.112@tcp [11847.651615] Lustre: server umount MGS complete [11857.689281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11858.071143] LustreError: 395786:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11858.089347] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11858.097263] Lustre: Skipped 1 previous similar message [11858.180606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11862.131472] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [11868.642604] LustreError: 395786:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11868.658591] LustreError: 395786:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11870.309282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11870.588165] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11875.196474] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [11881.660193] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11886.877340] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11888.530255] Lustre: Permanently reactivating lustre-OST0000 [11888.542575] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11888.553895] Lustre: Skipped 2 previous similar messages [11914.719510] LustreError: 397524:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11923.887515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11924.219673] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11924.223934] Lustre: Skipped 1 previous similar message [11930.944520] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [11938.549428] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11944.930641] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11944.933261] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11944.936815] LustreError: Skipped 1 previous similar message [11944.940035] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11944.957119] Lustre: Skipped 1 previous similar message [11945.836371] Lustre: server umount lustre-OST0000 complete [11952.610119] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11952.634970] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11955.170059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11955.188239] Lustre: Skipped 3 previous similar messages [11965.407630] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11965.554656] Lustre: server umount lustre-MDT0000 complete [11969.054924] LustreError: 395768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787586582 with bad export cookie 2863782722579275743 [11969.060538] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11969.073471] LustreError: 395768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11969.443702] Lustre: server umount lustre-MDT0001 complete [11979.078598] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [11981.533288] Key type lgssc unregistered [11981.809905] LNet: 400154:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11981.814772] LNetError: 400154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11982.893743] LNet: Removed LNI 192.168.206.112@tcp [11983.731768] Key type .llcrypt unregistered [11983.734133] Key type ._llcrypt unregistered [11999.528144] Key type ._llcrypt registered [11999.530166] Key type .llcrypt registered [11999.621273] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [12013.003228] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [12013.904189] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12013.997761] alg: No test for adler32 (adler32-zlib) [12015.118860] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [12015.431100] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [12017.135153] Key type lgssc registered [12017.951783] Lustre: Echo OBD driver; http://www.lustre.org/ [12049.289290] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12049.307431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12050.559197] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12050.590437] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12050.675100] Lustre: lustre-MDT0000: new disk, initializing [12050.783894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12050.805126] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12054.828694] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [12065.331443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12065.408429] Lustre: 403311:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12065.473259] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12065.476714] Lustre: Skipped 1 previous similar message [12065.538830] Lustre: lustre-MDT0001: new disk, initializing [12065.616302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12065.661864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12065.681936] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12070.159496] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [12079.401718] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12083.909038] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12090.016883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12090.355770] Lustre: lustre-OST0000: new disk, initializing [12090.360753] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12090.369774] Lustre: 405281:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12090.474538] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12096.616475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12096.643161] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12096.742122] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12096.779886] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [12106.889640] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12112.761817] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12113.066995] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12118.467302] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12118.800517] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12122.594536] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12122.596711] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12122.640816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12127.598088] Lustre: server umount lustre-OST0000 complete [12132.321829] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12132.330510] LustreError: Skipped 1 previous similar message [12132.335718] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12132.356689] Lustre: Skipped 1 previous similar message [12132.373943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12132.383403] Lustre: Skipped 1 previous similar message [12137.663929] Lustre: server umount lustre-MDT0000 complete [12141.601088] LustreError: 403304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787586754 with bad export cookie 13068986997342327968 [12141.602844] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12141.615837] LustreError: 403304:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12142.005628] Lustre: server umount lustre-MDT0001 complete [12149.479343] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12151.310449] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 11:52:43 (1787586763) [12157.916296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12158.310630] LustreError: 407740:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12158.334356] LustreError: 407740:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12158.413519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12161.875822] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [12163.555138] LustreError: 407741:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12168.674562] LustreError: 407740:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12168.704944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12168.946031] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12172.813190] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [12180.422526] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12186.227181] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12193.071537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12193.448312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12199.260916] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [12205.569587] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12207.113176] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12209.336033] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12214.251582] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12214.258079] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12214.268843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12214.268850] Lustre: Skipped 1 previous similar message [12214.285279] Lustre: Skipped 2 previous similar messages [12216.245284] Lustre: server umount lustre-OST0000 complete [12220.386868] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12220.396369] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12220.409453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12220.412136] Lustre: Skipped 1 previous similar message [12224.482964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12224.484393] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12226.246783] Lustre: server umount lustre-MDT0000 complete [12229.603876] LustreError: 408803:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12229.642641] LustreError: 408803:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12229.991244] LustreError: 407722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787586843 with bad export cookie 13068986997342328857 [12230.008716] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12230.019160] LustreError: 407722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12230.422494] Lustre: server umount lustre-MDT0001 complete [12241.449199] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [12243.986457] Key type lgssc unregistered [12244.284388] LNet: 411467:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12244.297159] LNetError: 411467:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12244.326112] LNet: Removed LNI 192.168.206.112@tcp [12245.132142] Key type .llcrypt unregistered [12245.134654] Key type ._llcrypt unregistered [12268.375903] Key type ._llcrypt registered [12268.379878] Key type .llcrypt registered [12268.498662] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [12268.804310] Key type .llcrypt unregistered [12268.810137] Key type ._llcrypt unregistered [12286.159407] Key type ._llcrypt registered [12286.164214] Key type .llcrypt registered [12286.322987] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [12287.534186] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12287.608634] alg: No test for adler32 (adler32-zlib) [12288.758609] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [12289.025936] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [12290.711223] Key type lgssc registered [12292.113805] Lustre: Echo OBD driver; http://www.lustre.org/ [12300.821861] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12300.839422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12302.328743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12307.014326] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [12312.548197] LustreError: 413692:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12312.563441] LustreError: 413692:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12315.507434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12315.808072] LustreError: 413693:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12315.836180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12321.249265] LustreError: 413692:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12321.275994] LustreError: 413692:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12322.471723] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [12326.371426] LustreError: 413688:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12326.386781] LustreError: 413688:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12327.409602] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12329.678842] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12332.530747] LustreError: 413692:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12332.560579] LustreError: 413692:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12336.106306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12336.421132] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12340.766948] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [12350.601671] Lustre: server umount lustre-OST0000 complete [12356.072553] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12356.084719] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12356.112903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12357.090486] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12357.092592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12357.112098] Lustre: Skipped 1 previous similar message [12362.208879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12362.217760] Lustre: Skipped 1 previous similar message [12364.334132] Lustre: server umount lustre-MDT0000 complete [12367.328682] LustreError: 413688:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12367.356846] LustreError: 413688:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12368.161556] LustreError: 415437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787586981 with bad export cookie 7874007770847984630 [12368.162576] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12368.177133] LustreError: 415437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12378.801029] Lustre: server umount lustre-MDT0001 complete [12389.987899] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [12392.544979] Key type lgssc unregistered [12392.854695] LNet: 416847:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12392.860479] LNetError: 416847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12392.880507] LNet: Removed LNI 192.168.206.112@tcp [12393.640100] Key type .llcrypt unregistered [12393.642102] Key type ._llcrypt unregistered [12418.295666] Key type ._llcrypt registered [12418.297802] Key type .llcrypt registered [12418.415931] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [12419.689682] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12419.718130] alg: No test for adler32 (adler32-zlib) [12420.717467] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [12420.964255] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [12422.647607] Key type lgssc registered [12423.745880] Lustre: Echo OBD driver; http://www.lustre.org/ [12430.833788] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12430.856905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12446.368294] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12451.487387] LustreError: 418237:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.112@tcp: failed processing log, type 4: rc = -110 [12478.177140] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12483.962890] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [12491.045766] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12497.292904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12497.751838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12497.825248] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12502.342294] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [12503.012259] LustreError: 418263:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12503.049878] LustreError: 418263:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12508.132363] LustreError: 418262:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12510.872836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12511.362353] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12515.080913] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [12522.927692] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12529.091454] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12536.801279] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12536.809811] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12536.823609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12538.295435] Lustre: server umount lustre-OST0000 complete [12541.925639] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12541.932952] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12541.952704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12544.494483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12544.504816] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12549.603836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12549.606688] Lustre: Skipped 2 previous similar messages [12551.733278] Lustre: server umount lustre-MDT0000 complete [12554.721463] LustreError: 419279:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12554.736572] LustreError: 419279:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12554.868250] LustreError: 418244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787587167 with bad export cookie 16754963137100801282 [12554.869733] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12554.875521] LustreError: 418244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12555.181136] Lustre: server umount lustre-MDT0001 complete [12564.932764] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [12567.542619] Key type lgssc unregistered [12567.794989] LNet: 421808:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12567.807592] LNetError: 421808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12567.829220] LNet: Removed LNI 192.168.206.112@tcp [12568.769514] Key type .llcrypt unregistered [12568.774617] Key type ._llcrypt unregistered [12596.125219] Key type ._llcrypt registered [12596.132559] Key type .llcrypt registered [12596.242366] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [12597.563209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12597.774097] alg: No test for adler32 (adler32-zlib) [12599.017298] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [12599.393595] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [12601.159466] Key type lgssc registered [12602.526504] Lustre: Echo OBD driver; http://www.lustre.org/ [12611.161678] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12611.197310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12612.682647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12617.155425] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [12622.819399] LustreError: 423226:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12622.842976] LustreError: 423226:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12625.441149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12625.851908] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12630.271638] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [12638.203107] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12646.409436] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12655.365808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12655.865944] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12657.932966] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12663.677622] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [12673.165845] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12680.677693] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12680.681896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12680.682051] Lustre: Skipped 2 previous similar messages [12680.694572] Lustre: Skipped 3 previous similar messages [12683.804957] Lustre: server umount lustre-MDT0000 complete [12685.761119] LustreError: 424838:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12685.827256] LustreError: 424838:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12686.709986] LustreError: 423208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.12@tcp arrived at 1787587299 with bad export cookie 4968649887869418813 [12686.732420] LustreError: 423208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12689.102720] LustreError: 423961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787587302 with bad export cookie 4968649887869417917 [12689.119445] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12689.121696] LustreError: 423961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12690.916251] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12690.916820] LustreError: 423954:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12690.917521] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12690.936736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12690.956384] LustreError: 423954:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12694.944214] LustreError: 423227:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12694.947904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12694.967275] LustreError: 423227:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12695.413406] Lustre: server umount lustre-OST0000 complete [12699.043418] LustreError: 423954:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12699.063659] LustreError: 423954:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12701.883987] Lustre: server umount lustre-MDT0001 complete [12713.205670] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [12715.996847] Key type lgssc unregistered [12716.388725] LNet: 426926:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12716.398749] LNetError: 426926:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12716.416974] LNet: Removed LNI 192.168.206.112@tcp [12717.439932] Key type .llcrypt unregistered [12717.442757] Key type ._llcrypt unregistered [12741.042168] Key type ._llcrypt registered [12741.044801] Key type .llcrypt registered [12741.162756] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [12742.266923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12742.372519] alg: No test for adler32 (adler32-zlib) [12743.389774] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [12743.571384] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [12745.199181] Key type lgssc registered [12746.190610] Lustre: Echo OBD driver; http://www.lustre.org/ [12753.635775] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12753.662944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12755.163292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12759.419372] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [12765.674837] LustreError: 428343:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12765.693207] LustreError: 428343:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12766.834727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12767.185219] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12771.189417] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [12778.205863] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12782.984477] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12788.508175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12788.864628] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12794.173655] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [12800.125083] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12801.968864] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12805.610725] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12805.638391] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12805.643252] Lustre: Skipped 1 previous similar message [12808.676814] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.12@tcp (stopping) [12810.720297] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12810.730622] Lustre: Skipped 1 previous similar message [12811.919066] Lustre: server umount lustre-MDT0001 complete [12815.842129] LustreError: 428344:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12815.871113] LustreError: 428344:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12820.962067] LustreError: 428343:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12820.962578] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12820.962715] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12820.962721] Lustre: Skipped 1 previous similar message [12820.963407] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12823.807561] Lustre: server umount lustre-OST0000 complete [12826.085675] LustreError: 428344:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12827.623636] Lustre: server umount lustre-MDT0000 complete [12842.169842] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [12844.645839] Key type lgssc unregistered [12844.888773] LNet: 432030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12844.907693] LNetError: 432030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12844.929533] LNet: Removed LNI 192.168.206.112@tcp [12845.817805] Key type .llcrypt unregistered [12845.825169] Key type ._llcrypt unregistered [12870.078737] Key type ._llcrypt registered [12870.081193] Key type .llcrypt registered [12870.160887] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [12882.681873] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [12883.506837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12883.537714] alg: No test for adler32 (adler32-zlib) [12884.553952] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [12884.789299] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [12886.455403] Key type lgssc registered [12887.482085] Lustre: Echo OBD driver; http://www.lustre.org/ [12923.593051] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [12940.551885] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12940.577834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12941.901277] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12941.974205] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12942.085360] Lustre: lustre-MDT0000: new disk, initializing [12942.194337] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12942.255201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12947.222438] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [12964.665274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12965.048640] Lustre: lustre-OST0000: new disk, initializing [12965.053292] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12965.062624] Lustre: Skipped 1 previous similar message [12965.068258] Lustre: 437064:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12965.151725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12971.161360] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [12974.133523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12974.146427] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12974.190478] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12982.341918] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12999.673315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12999.855422] Lustre: 436045:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12999.928264] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13000.048068] Lustre: lustre-MDT0001: new disk, initializing [13000.208947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13000.253254] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13000.274962] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13005.986215] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [13018.955683] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13019.431646] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13027.158907] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13027.563381] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13034.999358] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13035.000922] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13035.034740] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13035.055078] Lustre: Skipped 1 previous similar message [13035.080486] Lustre: Skipped 1 previous similar message [13040.104372] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13040.117947] Lustre: Skipped 2 previous similar messages [13040.482936] Lustre: server umount lustre-MDT0001 complete [13050.344156] LustreError: 436051:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13050.386693] LustreError: 436051:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [13055.466109] LustreError: 437776:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13055.496791] LustreError: 437776:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13060.577150] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13060.580599] LustreError: 436057:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13060.613314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13064.696828] Lustre: server umount lustre-OST0000 complete [13065.700660] LustreError: 438773:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13075.355498] Lustre: server umount lustre-MDT0000 complete [13091.792988] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [13094.885806] Key type lgssc unregistered [13095.251693] LNet: 440814:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13095.259553] LNetError: 440814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13096.294769] LNet: Removed LNI 192.168.206.112@tcp [13097.612489] Key type .llcrypt unregistered [13097.614284] Key type ._llcrypt unregistered [13136.290606] Key type ._llcrypt registered [13136.295598] Key type .llcrypt registered [13136.411638] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [13136.745414] Key type .llcrypt unregistered [13136.749197] Key type ._llcrypt unregistered [13153.078329] Key type ._llcrypt registered [13153.079696] Key type .llcrypt registered [13153.161859] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [13154.029029] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13154.151122] alg: No test for adler32 (adler32-zlib) [13155.228133] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [13155.450360] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [13157.119669] Key type lgssc registered [13158.389627] Lustre: Echo OBD driver; http://www.lustre.org/ [13185.775970] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13185.830398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13187.163340] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13187.193970] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13187.318683] Lustre: lustre-MDT0000: new disk, initializing [13187.417126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13187.444730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13192.371755] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [13203.234380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13203.387230] Lustre: 444199:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13203.480035] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13203.486271] Lustre: Skipped 1 previous similar message [13203.599754] Lustre: lustre-MDT0001: new disk, initializing [13203.731115] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13203.754389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13203.763522] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13208.380873] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [13217.714907] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13223.553666] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13230.534629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13230.762421] Lustre: lustre-OST0000: new disk, initializing [13230.766471] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13230.772371] Lustre: 446169:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13230.837990] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13237.324326] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [13240.395478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13240.418132] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13240.565699] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13247.338245] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13255.136727] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13255.148172] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13255.180448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13256.171846] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13256.172133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13256.202972] Lustre: Skipped 1 previous similar message [13260.520727] Lustre: server umount lustre-MDT0000 complete [13264.891904] LustreError: 445004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787587878 with bad export cookie 6724588488953008809 [13264.894058] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13264.908236] LustreError: 445004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13265.396523] Lustre: server umount lustre-MDT0001 complete [13279.981209] Lustre: server umount lustre-OST0000 complete [13292.647065] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 12:11:44 (1787587904) [13301.466966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13302.055606] LustreError: 448198:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13302.073544] LustreError: 448198:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13302.144877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13307.199295] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [13307.372574] LustreError: 448199:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13312.484055] LustreError: 448198:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13317.564358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13317.605506] LustreError: 448199:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13318.120486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13323.636417] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [13330.997837] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13337.123325] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13356.728035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13357.018954] Lustre: Found index 0 for lustre-OST0000, updating log [13357.300629] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13363.700624] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [13372.125464] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13373.692290] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13384.166438] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13384.168273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13384.196738] Lustre: Skipped 2 previous similar messages [13384.238090] Lustre: Skipped 1 previous similar message [13389.969753] Lustre: server umount lustre-OST0000 complete [13407.201227] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13407.230345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13407.234351] Lustre: Skipped 1 previous similar message [13407.254454] Lustre: Skipped 3 previous similar messages [13411.361524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13411.368959] Lustre: Skipped 1 previous similar message [13413.162230] Lustre: server umount lustre-MDT0000 complete [13417.440579] LustreError: 449149:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13417.469666] LustreError: 449149:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13417.807167] LustreError: 448181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787588030 with bad export cookie 6724588488953010209 [13417.807969] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13417.830325] LustreError: 448181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13418.434623] Lustre: server umount lustre-MDT0001 complete [13419.679228] LNet: 1 local NIs in recovery (showing 1): 192.168.206.112@tcp [13428.267539] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 12:13:59 (1787588039) [14048.158223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14048.813687] LustreError: 454612:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14048.891755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14054.161839] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [14054.370339] LustreError: 454613:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14058.468879] LustreError: 454612:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14064.566270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14064.608778] LustreError: 454612:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14064.646184] LustreError: 454612:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14065.335205] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14072.014616] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [14081.724345] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14088.217705] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14109.230500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14109.503796] Lustre: Found index 0 for lustre-OST0000, updating log [14109.839979] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14110.881599] LNet: 1 local NIs in recovery (showing 1): 192.168.206.112@tcp [14117.294395] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [14125.077097] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14128.260701] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14128.283902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14128.291219] Lustre: Skipped 2 previous similar messages [14133.431636] Lustre: server umount lustre-OST0000 complete [14138.275960] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14138.284469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14138.289677] Lustre: Skipped 1 previous similar message [14138.313568] Lustre: Skipped 1 previous similar message [14140.324945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14140.327496] Lustre: Skipped 2 previous similar messages [14143.592591] Lustre: server umount lustre-MDT0000 complete [14144.425617] LustreError: 457167:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14144.441733] LustreError: 457167:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14147.338258] LustreError: 454592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787588760 with bad export cookie 6724588488953011294 [14147.343201] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14147.348301] LustreError: 454592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14147.601779] Lustre: server umount lustre-MDT0001 complete [14170.272186] LNet: 1 local NIs in recovery (showing 1): 192.168.206.112@tcp [14180.083545] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [14191.069398] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [14229.895154] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 12:27:21 (1787588841) [14246.203310] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [14248.621761] Key type lgssc unregistered [14248.885236] LNet: 461230:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14248.892912] LNetError: 461230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14370.791349] LNet: Removed LNI 192.168.206.112@tcp [14371.890946] Key type .llcrypt unregistered [14371.893313] Key type ._llcrypt unregistered [14389.245469] Key type ._llcrypt registered [14389.248278] Key type .llcrypt registered [14389.349291] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [14390.603379] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14390.782663] alg: No test for adler32 (adler32-zlib) [14391.880465] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [14392.122186] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [14393.799210] Key type lgssc registered [14394.867706] Lustre: Echo OBD driver; http://www.lustre.org/ [14399.115977] LNet: Added LNI 192.168.206.112@tcp100 [8/256/0/180] [14399.866341] LNet: Added LNI 192.168.206.112@tcp101 [8/256/0/180] [14400.990996] LNet: Added LNI 192.168.206.112@tcp102 [8/256/0/180] [14402.230150] LNet: Added LNI 192.168.206.112@tcp103 [8/256/0/180] [14403.312603] LNet: Added LNI 192.168.206.112@tcp104 [8/256/0/180] [14404.352862] LNet: Added LNI 192.168.206.112@tcp105 [8/256/0/180] [14405.246360] LNet: Added LNI 192.168.206.112@tcp106 [8/256/0/180] [14406.175508] LNet: Added LNI 192.168.206.112@tcp107 [8/256/0/180] [14407.173868] LNet: Added LNI 192.168.206.112@tcp108 [8/256/0/180] [14408.129034] LNet: Added LNI 192.168.206.112@tcp109 [8/256/0/180] [14409.015191] LNet: Added LNI 192.168.206.112@tcp110 [8/256/0/180] [14409.946240] LNet: Added LNI 192.168.206.112@tcp111 [8/256/0/180] [14410.783494] LNet: Added LNI 192.168.206.112@tcp112 [8/256/0/180] [14411.530453] LNet: Added LNI 192.168.206.112@tcp113 [8/256/0/180] [14412.386889] LNet: Added LNI 192.168.206.112@tcp114 [8/256/0/180] [14413.171270] LNet: Added LNI 192.168.206.112@tcp115 [8/256/0/180] [14413.781718] LNet: Added LNI 192.168.206.112@tcp116 [8/256/0/180] [14414.482048] LNet: Added LNI 192.168.206.112@tcp117 [8/256/0/180] [14415.198129] LNet: Added LNI 192.168.206.112@tcp118 [8/256/0/180] [14416.010397] LNet: Added LNI 192.168.206.112@tcp119 [8/256/0/180] [14416.901941] LNet: Added LNI 192.168.206.112@tcp120 [8/256/0/180] [14417.716512] LNet: Added LNI 192.168.206.112@tcp121 [8/256/0/180] [14418.533206] LNet: Added LNI 192.168.206.112@tcp122 [8/256/0/180] [14419.383485] LNet: Added LNI 192.168.206.112@tcp123 [8/256/0/180] [14420.163526] LNet: Added LNI 192.168.206.112@tcp124 [8/256/0/180] [14421.026540] LNet: Added LNI 192.168.206.112@tcp125 [8/256/0/180] [14421.956738] LNet: Added LNI 192.168.206.112@tcp126 [8/256/0/180] [14422.847569] LNet: Added LNI 192.168.206.112@tcp127 [8/256/0/180] [14423.769586] LNet: Added LNI 192.168.206.112@tcp128 [8/256/0/180] [14424.708407] LNet: Added LNI 192.168.206.112@tcp129 [8/256/0/180] [14425.509588] LNet: Added LNI 192.168.206.112@tcp130 [8/256/0/180] [14426.456836] LNet: Added LNI 192.168.206.112@tcp131 [8/256/0/180] [14427.291862] LNet: Added LNI 192.168.206.112@tcp132 [8/256/0/180] [14428.120539] LNet: Added LNI 192.168.206.112@tcp133 [8/256/0/180] [14429.022580] LNet: Added LNI 192.168.206.112@tcp134 [8/256/0/180] [14429.802481] LNet: Added LNI 192.168.206.112@tcp135 [8/256/0/180] [14437.370417] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14437.403858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14438.536374] Lustre: 464263:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.112@tcp: can't add NID 192.168.206.112@tcp130: rc = -75 [14438.635024] LNetError: 461723:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.112 [14438.655814] LNetError: Protocol error connecting to 192.168.206.112@tcp122 at host 192.168.206.112:988: is it running a compatible version of Lustre? [14438.744852] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14438.775419] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14438.838525] Lustre: lustre-MDT0000: new disk, initializing [14438.911182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14438.930175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14439.263392] LNet: 4 local NIs in recovery (showing 4): 192.168.206.112@tcp122, 192.168.206.112@tcp126, 192.168.206.112@tcp128, 192.168.206.112@tcp129 [14439.264508] LNetError: 461723:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.112 [14439.288127] LNetError: 461723:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 5 previous similar messages [14439.303859] LNetError: Protocol error connecting to 192.168.206.112@tcp106 at host 192.168.206.112:988: is it running a compatible version of Lustre? [14439.321243] LNetError: Skipped 5 previous similar messages [14443.825692] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [14454.325031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14454.485464] Lustre: 464279:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14454.536323] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14454.539654] Lustre: Skipped 1 previous similar message [14454.599195] Lustre: lustre-MDT0001: new disk, initializing [14454.633211] Lustre: 465082:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.112@tcp: can't add NID 192.168.206.112@tcp130: rc = -75 [14454.644397] Lustre: 465082:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14454.777858] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14454.830213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14454.851487] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14459.418306] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [14461.980989] Lustre: 465447:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.112@tcp: can't add NID 192.168.206.112@tcp130: rc = -75 [14461.999730] Lustre: 465447:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14468.583324] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14473.056654] LNetError: 464284:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.112 [14473.072976] LNetError: Protocol error connecting to 192.168.206.112@tcp106 at host 192.168.206.112:988: is it running a compatible version of Lustre? [14474.388672] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14476.129793] LNetError: 461724:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.112 [14476.135730] LNetError: Protocol error connecting to 192.168.206.112@tcp125 at host 192.168.206.112:988: is it running a compatible version of Lustre? [14481.143936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14481.435169] Lustre: lustre-OST0000: new disk, initializing [14481.437887] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14481.443771] Lustre: 466254:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14481.519269] Lustre: 466259:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.112@tcp: can't add NID 192.168.206.112@tcp130: rc = -75 [14481.526745] Lustre: 466259:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14481.580857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14486.369478] LNetError: 464283:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.112 [14486.382087] LNetError: Protocol error connecting to 192.168.206.112@tcp106 at host 192.168.206.112:988: is it running a compatible version of Lustre? [14488.009478] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [14488.105869] Lustre: 466611:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.112@tcp: can't add NID 192.168.206.112@tcp130: rc = -75 [14488.114702] Lustre: 466611:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14488.182627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14488.198727] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14488.370708] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14498.118944] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14499.689778] LNetError: 464292:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.112 [14499.708344] LNetError: 464292:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [14499.714301] LNetError: Protocol error connecting to 192.168.206.112@tcp106 at host 192.168.206.112:988: is it running a compatible version of Lustre? [14499.746189] LNetError: Skipped 1 previous similar message [14509.027444] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14509.037456] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14509.055949] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14509.064851] Lustre: Skipped 1 previous similar message [14513.676664] Lustre: server umount lustre-OST0000 complete [14519.264987] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14519.287031] Lustre: Skipped 1 previous similar message [14519.298848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14524.159220] Lustre: server umount lustre-MDT0000 complete [14524.327116] LNetError: 461723:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.112 [14524.339816] LNetError: 461723:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 2 previous similar messages [14524.351597] LNetError: Protocol error connecting to 192.168.206.112@tcp106 at host 192.168.206.112:988: is it running a compatible version of Lustre? [14524.363650] LNetError: Skipped 2 previous similar messages [14528.263172] LustreError: 464271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787589141 with bad export cookie 15381551362461700309 [14528.266283] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14528.277166] LustreError: 464271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14528.603530] Lustre: server umount lustre-MDT0001 complete [14539.470372] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [14542.702306] Key type lgssc unregistered [14543.092660] LNet: 468170:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14543.100147] LNetError: 468170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14543.106427] LNet: Removed LNI 192.168.206.112@tcp [14544.159725] LNet: Removed LNI 192.168.206.112@tcp100 [14544.162405] LNetError: 468170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14544.182985] LNetError: 468170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14544.192254] LNet: Removed LNI 192.168.206.112@tcp101 [14545.249559] LNet: Removed LNI 192.168.206.112@tcp102 [14545.251897] LNetError: 468170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14545.266402] LNetError: 468170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14545.271298] LNet: Removed LNI 192.168.206.112@tcp103 [14546.335163] LNet: Removed LNI 192.168.206.112@tcp104 [14547.359155] LNet: Removed LNI 192.168.206.112@tcp105 [14547.365219] LNetError: 468170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14547.376550] LNetError: 468170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14547.384809] LNet: Removed LNI 192.168.206.112@tcp106 [14548.447493] LNet: Removed LNI 192.168.206.112@tcp107 [14549.471433] LNet: Removed LNI 192.168.206.112@tcp108 [14550.495182] LNet: Removed LNI 192.168.206.112@tcp109 [14551.519756] LNet: Removed LNI 192.168.206.112@tcp110 [14551.523878] LNetError: 468170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14551.546858] LNetError: 468170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14551.560510] LNet: Removed LNI 192.168.206.112@tcp111 [14552.609731] LNet: Removed LNI 192.168.206.112@tcp112 [14553.633236] LNet: Removed LNI 192.168.206.112@tcp113 [14554.657765] LNet: Removed LNI 192.168.206.112@tcp114 [14555.679325] LNet: Removed LNI 192.168.206.112@tcp115 [14556.703434] LNet: Removed LNI 192.168.206.112@tcp116 [14557.727849] LNet: Removed LNI 192.168.206.112@tcp117 [14558.751919] LNet: Removed LNI 192.168.206.112@tcp118 [14559.775499] LNet: Removed LNI 192.168.206.112@tcp119 [14559.782994] LNetError: 468170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14559.795079] LNetError: 468170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14559.802532] LNet: Removed LNI 192.168.206.112@tcp120 [14560.863449] LNet: Removed LNI 192.168.206.112@tcp121 [14561.887447] LNet: Removed LNI 192.168.206.112@tcp122 [14562.911222] LNet: Removed LNI 192.168.206.112@tcp123 [14563.936284] LNet: Removed LNI 192.168.206.112@tcp124 [14564.959690] LNet: Removed LNI 192.168.206.112@tcp125 [14565.983237] LNet: Removed LNI 192.168.206.112@tcp126 [14565.986970] LNet: Removed LNI 192.168.206.112@tcp127 [14567.007176] LNet: Removed LNI 192.168.206.112@tcp128 [14568.031137] LNet: Removed LNI 192.168.206.112@tcp129 [14568.034801] LNet: Removed LNI 192.168.206.112@tcp130 [14568.040972] LNet: Removed LNI 192.168.206.112@tcp131 [14568.044591] LNet: Removed LNI 192.168.206.112@tcp132 [14568.047408] LNet: Removed LNI 192.168.206.112@tcp133 [14568.050255] LNet: Removed LNI 192.168.206.112@tcp134 [14568.073713] LNet: Removed LNI 192.168.206.112@tcp135 [14568.947136] Key type .llcrypt unregistered [14568.951809] Key type ._llcrypt unregistered [14593.996220] Key type ._llcrypt registered [14593.997548] Key type .llcrypt registered [14594.099901] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [14609.371841] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [14610.555420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14610.584658] alg: No test for adler32 (adler32-zlib) [14611.710382] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [14611.975138] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [14613.647258] Key type lgssc registered [14614.917262] Lustre: Echo OBD driver; http://www.lustre.org/ [14653.042955] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 12:34:24 (1787589264) [14671.564276] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [14674.730758] Key type lgssc unregistered [14675.106538] LNet: 472278:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14675.113919] LNetError: 472278:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14675.135057] LNet: Removed LNI 192.168.206.112@tcp [14676.349677] Key type .llcrypt unregistered [14676.353658] Key type ._llcrypt unregistered [14692.729436] Key type ._llcrypt registered [14692.734429] Key type .llcrypt registered [14692.905849] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [14693.983939] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14694.018306] alg: No test for adler32 (adler32-zlib) [14695.055564] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [14695.343381] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [14697.095169] Key type lgssc registered [14698.165052] Lustre: Echo OBD driver; http://www.lustre.org/ [14708.896126] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14708.914740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14710.282867] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14710.353385] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14710.522355] Lustre: lustre-MDT0000: new disk, initializing [14710.690424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14710.747260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14716.451901] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [14728.724468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14728.829924] Lustre: 473493:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14728.876792] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14728.881033] Lustre: Skipped 1 previous similar message [14728.937233] Lustre: lustre-MDT0001: new disk, initializing [14729.030941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14729.076246] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14729.088234] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14734.627569] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [14745.135825] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14751.528581] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14759.576911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14759.869755] Lustre: lustre-OST0000: new disk, initializing [14759.874564] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14759.883988] Lustre: 475464:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14759.991171] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14766.684663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14766.699256] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14766.716682] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [14766.825841] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14775.473962] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14780.641705] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing lnet_if_list [14786.214582] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14786.246312] LNet: Added LNI 192.168.206.112@tcp10 [8/256/0/180] [14791.621947] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14791.644470] LNet: Added LNI 192.168.206.112@tcp11 [8/256/0/180] [14796.114857] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14796.142520] LNet: Added LNI 192.168.206.112@tcp12 [8/256/0/180] [14801.399346] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14801.471454] LNet: Added LNI 192.168.206.112@tcp13 [8/256/0/180] [14806.405465] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14806.440573] LNet: Added LNI 192.168.206.112@tcp14 [8/256/0/180] [14811.423407] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14811.465719] LNet: Added LNI 192.168.206.112@tcp15 [8/256/0/180] [14817.111383] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14817.151622] LNet: Added LNI 192.168.206.112@tcp16 [8/256/0/180] [14822.559298] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14822.594611] LNet: Added LNI 192.168.206.112@tcp17 [8/256/0/180] [14828.018229] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14828.060806] LNet: Added LNI 192.168.206.112@tcp18 [8/256/0/180] [14834.114553] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14834.153568] LNet: Added LNI 192.168.206.112@tcp19 [8/256/0/180] [14842.242694] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14842.313107] LNet: Added LNI 192.168.206.112@tcp20 [8/256/0/180] [14848.769500] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14848.791125] LNet: Added LNI 192.168.206.112@tcp21 [8/256/0/180] [14854.773164] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14854.803781] LNet: Added LNI 192.168.206.112@tcp22 [8/256/0/180] [14860.844926] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14860.877551] LNet: Added LNI 192.168.206.112@tcp23 [8/256/0/180] [14867.246757] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14867.284238] LNet: Added LNI 192.168.206.112@tcp24 [8/256/0/180] [14872.653065] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14872.694946] LNet: Added LNI 192.168.206.112@tcp25 [8/256/0/180] [14878.025587] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14878.051194] LNet: Added LNI 192.168.206.112@tcp26 [8/256/0/180] [14883.849383] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14883.891854] LNet: Added LNI 192.168.206.112@tcp27 [8/256/0/180] [14889.557312] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14889.582642] LNet: Added LNI 192.168.206.112@tcp28 [8/256/0/180] [14895.479711] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14895.505928] LNet: Added LNI 192.168.206.112@tcp29 [8/256/0/180] [14901.073569] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14901.100551] LNet: Added LNI 192.168.206.112@tcp30 [8/256/0/180] [14907.115589] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14907.171631] LNet: Added LNI 192.168.206.112@tcp31 [8/256/0/180] [14914.118761] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14914.166662] LNet: Added LNI 192.168.206.112@tcp32 [8/256/0/180] [14921.576272] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14921.637872] LNet: Added LNI 192.168.206.112@tcp33 [8/256/0/180] [14929.894954] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14929.936517] LNet: Added LNI 192.168.206.112@tcp34 [8/256/0/180] [14938.114176] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14938.180969] LNet: Added LNI 192.168.206.112@tcp35 [8/256/0/180] [14946.824616] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14946.877938] LNet: Added LNI 192.168.206.112@tcp36 [8/256/0/180] [14958.506059] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14958.560269] LNet: Added LNI 192.168.206.112@tcp37 [8/256/0/180] [14967.326138] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14967.351108] LNet: Added LNI 192.168.206.112@tcp38 [8/256/0/180] [14974.062976] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14974.095880] LNet: Added LNI 192.168.206.112@tcp39 [8/256/0/180] [14981.982417] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14982.017405] LNet: Added LNI 192.168.206.112@tcp40 [8/256/0/180] [14990.286722] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14990.328202] LNet: Added LNI 192.168.206.112@tcp41 [8/256/0/180] [14997.825865] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14997.899033] LNet: Added LNI 192.168.206.112@tcp42 [8/256/0/180] [15007.083123] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15007.117217] LNet: Added LNI 192.168.206.112@tcp43 [8/256/0/180] [15015.941666] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15015.979530] LNet: Added LNI 192.168.206.112@tcp44 [8/256/0/180] [15026.435654] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15026.481902] LNet: Added LNI 192.168.206.112@tcp45 [8/256/0/180] [15035.696505] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15035.750990] LNet: Added LNI 192.168.206.112@tcp46 [8/256/0/180] [15046.149994] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15046.186963] LNet: Added LNI 192.168.206.112@tcp47 [8/256/0/180] [15055.508756] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15055.539973] LNet: Added LNI 192.168.206.112@tcp48 [8/256/0/180] [15063.509897] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15063.586267] LNet: Added LNI 192.168.206.112@tcp49 [8/256/0/180] [15072.124674] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15072.164517] LNet: Added LNI 192.168.206.112@tcp50 [8/256/0/180] [15090.661520] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15090.684144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15090.685802] Lustre: Skipped 1 previous similar message [15090.708412] Lustre: Skipped 2 previous similar messages [15095.781538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15095.795188] Lustre: Skipped 2 previous similar messages [15096.384729] Lustre: server umount lustre-MDT0000 complete [15100.245203] LustreError: 473484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787589713 with bad export cookie 3815936518146237330 [15100.259183] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15100.271148] LustreError: 473484:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15100.677985] Lustre: server umount lustre-MDT0001 complete [15117.279127] Lustre: 472765:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787589714/real 1787589714] req@00000000cabc9c3d x1874423246404608/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787589730 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15117.309248] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15117.327737] Lustre: Skipped 1 previous similar message [15120.598403] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [15122.400552] Lustre: 472767:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787589719/real 1787589719] req@00000000df2f0112 x1874423246404736/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787589735 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15126.559508] Lustre: 472767:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787589723/real 1787589723] req@0000000073657a2c x1874423246404992/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787589739 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15129.459717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15131.618480] Lustre: 472764:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787589728/real 1787589728] req@000000006a0a8d05 x1874423246405120/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787589744 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15144.991797] LustreError: 472763:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000012205ce2 x1874423246405888/t0(0) o250->MGC192.168.206.112@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15145.384230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15146.416492] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15150.569705] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [15151.592429] LustreError: 475457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15151.612575] LustreError: 475457:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [15156.708809] LustreError: 475456:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15159.204151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15159.571083] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15162.661041] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15163.958434] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [15172.003712] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15178.490320] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15200.737695] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15200.754629] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15200.773591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15201.766794] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15203.811886] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15203.824905] Lustre: Skipped 2 previous similar messages [15205.525829] Lustre: server umount lustre-OST0000 complete [15210.976524] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15210.990457] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15211.001873] Lustre: Skipped 1 previous similar message [15211.006399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15216.189705] Lustre: server umount lustre-MDT0000 complete [15219.201333] LustreError: 499712:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15219.234444] LustreError: 499712:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15220.945064] LustreError: 473485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787589834 with bad export cookie 3815936518146239913 [15220.946522] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15220.958681] LustreError: 473485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15221.398241] Lustre: server umount lustre-MDT0001 complete [15234.247484] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [15237.927400] Key type lgssc unregistered [15238.553021] LNet: 502513:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15238.565219] LNetError: 502513:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15238.577694] LNet: Removed LNI 192.168.206.112@tcp [15238.585549] LNet: Removed LNI 192.168.206.112@tcp10 [15238.590308] LNet: Removed LNI 192.168.206.112@tcp11 [15238.594073] LNet: Removed LNI 192.168.206.112@tcp12 [15238.613903] LNet: Removed LNI 192.168.206.112@tcp13 [15238.618869] LNet: Removed LNI 192.168.206.112@tcp14 [15238.624898] LNet: Removed LNI 192.168.206.112@tcp15 [15238.628433] LNet: Removed LNI 192.168.206.112@tcp16 [15238.631678] LNet: Removed LNI 192.168.206.112@tcp17 [15238.636793] LNet: Removed LNI 192.168.206.112@tcp18 [15238.640173] LNet: Removed LNI 192.168.206.112@tcp19 [15238.642626] LNet: Removed LNI 192.168.206.112@tcp20 [15238.645880] LNet: Removed LNI 192.168.206.112@tcp21 [15238.648789] LNet: Removed LNI 192.168.206.112@tcp22 [15238.651348] LNet: Removed LNI 192.168.206.112@tcp23 [15238.654677] LNet: Removed LNI 192.168.206.112@tcp24 [15238.656666] LNet: Removed LNI 192.168.206.112@tcp25 [15238.659422] LNet: Removed LNI 192.168.206.112@tcp26 [15238.662515] LNet: Removed LNI 192.168.206.112@tcp27 [15238.665478] LNet: Removed LNI 192.168.206.112@tcp28 [15238.668471] LNet: Removed LNI 192.168.206.112@tcp29 [15238.671388] LNet: Removed LNI 192.168.206.112@tcp30 [15238.674490] LNet: Removed LNI 192.168.206.112@tcp31 [15238.677482] LNet: Removed LNI 192.168.206.112@tcp32 [15238.680158] LNet: Removed LNI 192.168.206.112@tcp33 [15238.682417] LNet: Removed LNI 192.168.206.112@tcp34 [15238.685513] LNet: Removed LNI 192.168.206.112@tcp35 [15238.688285] LNet: Removed LNI 192.168.206.112@tcp36 [15238.691542] LNet: Removed LNI 192.168.206.112@tcp37 [15238.694621] LNet: Removed LNI 192.168.206.112@tcp38 [15238.697518] LNet: Removed LNI 192.168.206.112@tcp39 [15238.700336] LNet: Removed LNI 192.168.206.112@tcp40 [15238.703249] LNet: Removed LNI 192.168.206.112@tcp41 [15238.707174] LNet: Removed LNI 192.168.206.112@tcp42 [15238.710692] LNet: Removed LNI 192.168.206.112@tcp43 [15238.713804] LNet: Removed LNI 192.168.206.112@tcp44 [15238.715856] LNet: Removed LNI 192.168.206.112@tcp45 [15238.718226] LNet: Removed LNI 192.168.206.112@tcp46 [15238.720750] LNet: Removed LNI 192.168.206.112@tcp47 [15238.723303] LNet: Removed LNI 192.168.206.112@tcp48 [15238.725977] LNet: Removed LNI 192.168.206.112@tcp49 [15238.734693] LNet: Removed LNI 192.168.206.112@tcp50 [15239.497647] Key type .llcrypt unregistered [15239.503692] Key type ._llcrypt unregistered [15262.959806] Key type ._llcrypt registered [15262.962462] Key type .llcrypt registered [15263.059643] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [15280.637949] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [15282.107112] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15282.389747] alg: No test for adler32 (adler32-zlib) [15283.837770] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [15284.251712] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [15285.952343] Key type lgssc registered [15287.435709] Lustre: Echo OBD driver; http://www.lustre.org/ [15323.920563] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15323.979366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15325.327215] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15325.350793] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15325.416953] Lustre: lustre-MDT0000: new disk, initializing [15325.558525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15325.585981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15330.101540] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [15340.511854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15340.662165] Lustre: 505712:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15340.727244] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15340.730642] Lustre: Skipped 1 previous similar message [15340.812950] Lustre: lustre-MDT0001: new disk, initializing [15340.982725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15341.027710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15341.037890] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15346.427271] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [15355.918054] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15362.568889] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15370.932067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15371.144907] Lustre: *** cfs_fail_loc=90f, val=0*** [15371.222417] Lustre: lustre-OST0000: new disk, initializing [15371.225475] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15371.233597] Lustre: 507731:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15371.333678] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15377.157903] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [15380.589441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15380.601257] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15380.707829] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15387.304406] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15401.443703] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15401.446771] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15401.447210] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15401.457374] Lustre: Skipped 1 previous similar message [15403.905588] Lustre: server umount lustre-OST0000 complete [15411.683194] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15411.686444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15411.700485] Lustre: Skipped 1 previous similar message [15411.709159] Lustre: Skipped 2 previous similar messages [15414.266974] Lustre: server umount lustre-MDT0000 complete [15418.785992] LustreError: 505703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787590031 with bad export cookie 15805409065684182729 [15418.803688] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15418.815672] LustreError: 505703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15419.260202] Lustre: server umount lustre-MDT0001 complete [15431.726446] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [15434.422500] Key type lgssc unregistered [15434.728811] LNet: 509641:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15434.742686] LNetError: 509641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15434.762674] LNet: Removed LNI 192.168.206.112@tcp [15435.587842] Key type .llcrypt unregistered [15435.591332] Key type ._llcrypt unregistered [15454.765654] Key type ._llcrypt registered [15454.767628] Key type .llcrypt registered [15454.869594] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [15455.704351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15455.768114] alg: No test for adler32 (adler32-zlib) [15456.825971] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [15457.089713] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [15458.759503] Key type lgssc registered [15459.793881] Lustre: Echo OBD driver; http://www.lustre.org/ [15467.009454] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15467.039916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15468.613905] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15473.280993] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [15478.766149] LustreError: 510919:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15478.783611] LustreError: 510919:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [15481.451518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15481.781838] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15485.704905] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [15492.651251] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15498.387657] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15505.181895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15505.715820] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15512.114962] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [15519.574881] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15530.422458] loop0: detected capacity change from 0 to 204800000 [15530.629412] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15530.640925] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15530.656810] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15536.546976] loop0: detected capacity change from 0 to 204800000 [15536.735722] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15536.744321] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15536.759050] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15541.768536] loop0: detected capacity change from 0 to 204800000 [15547.071982] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15547.202045] Lustre: fs15246-OST0000: new disk, initializing [15547.208699] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15547.216432] Lustre: 514508:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15547.269972] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15555.024067] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [15564.271967] loop1: detected capacity change from 0 to 204800000 [15570.171917] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15570.265913] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15570.314787] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15570.385825] Lustre: fs15246-MDT0000: new disk, initializing [15570.521560] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15570.584778] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15570.744650] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15571.810107] Lustre: 515629:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000076d0751f x1874424045287680/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787590200 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15574.596899] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [15576.875439] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15576.884732] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15576.934620] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15597.033709] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15597.052839] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15601.341746] Lustre: server umount fs15246-MDT0000 complete [15602.150523] LustreError: 510914:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15607.271348] LustreError: 510914:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15611.279234] Lustre: server umount fs15246-OST0000 complete [15622.629498] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15622.655978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15622.665746] Lustre: Skipped 1 previous similar message [15627.744740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15627.764058] Lustre: Skipped 1 previous similar message [15628.854930] Lustre: server umount lustre-OST0000 complete [15632.865331] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15632.887462] Lustre: Skipped 1 previous similar message [15632.896054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15632.901287] Lustre: Skipped 1 previous similar message [15637.991816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15638.006309] Lustre: Skipped 1 previous similar message [15639.092141] Lustre: server umount lustre-MDT0000 complete [15643.104701] LustreError: 511647:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15643.132847] LustreError: 511647:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15643.271263] LustreError: 510898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787590256 with bad export cookie 4257778217854280565 [15643.275883] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15643.285935] LustreError: 510898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15643.792807] Lustre: server umount lustre-MDT0001 complete [15654.969764] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [15657.982502] Key type lgssc unregistered [15658.281488] LNet: 518729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15658.291921] LNetError: 518729:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15658.313059] LNet: Removed LNI 192.168.206.112@tcp [15659.200762] Key type .llcrypt unregistered [15659.203028] Key type ._llcrypt unregistered [15678.173933] Key type ._llcrypt registered [15678.180274] Key type .llcrypt registered [15678.271322] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [15679.153704] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15679.195580] alg: No test for adler32 (adler32-zlib) [15680.302809] Lustre: Lustre: Build Version: 2.17.57_80_g1214e44 [15680.575261] LNet: Added LNI 192.168.206.112@tcp [8/256/0/180] [15682.255181] Key type lgssc registered [15683.304715] Lustre: Echo OBD driver; http://www.lustre.org/ [15722.785453] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [15733.612932] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [15774.244946] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15774.300584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15775.547918] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15775.584978] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15775.716786] Lustre: lustre-MDT0000: new disk, initializing [15775.825123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15775.856469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15781.162334] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [15791.562915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15791.693772] Lustre: 523203:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15791.784099] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15791.788673] Lustre: Skipped 1 previous similar message [15791.921841] Lustre: lustre-MDT0001: new disk, initializing [15792.043512] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15792.103581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15792.121715] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15797.045155] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [15807.044708] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15813.237444] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15820.536528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15820.732736] Lustre: lustre-OST0000: new disk, initializing [15820.736550] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15820.742060] Lustre: 525171:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15820.821786] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15826.327397] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_default_debug -1 all [15829.568495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15829.585691] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15829.701651] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15835.004598] Lustre: DEBUG MARKER: oleg612-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15839.712291] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15839.933228] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15844.695932] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15844.957571] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15850.468781] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15850.472785] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15850.473292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15850.488392] Lustre: Skipped 1 previous similar message [15855.590401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15855.600330] Lustre: Skipped 1 previous similar message [15860.707899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15860.717444] Lustre: Skipped 1 previous similar message [15861.215167] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15861.434088] Lustre: server umount lustre-OST0000 complete [15865.827721] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15865.829376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15865.856266] Lustre: Skipped 1 previous similar message [15865.867562] Lustre: Skipped 2 previous similar messages [15870.945251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15870.957864] Lustre: Skipped 1 previous similar message [15871.865642] Lustre: server umount lustre-MDT0000 complete [15876.852950] LustreError: 524007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787590489 with bad export cookie 13181365395097116440 [15876.864385] LustreError: MGC192.168.206.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15876.874832] LustreError: 524007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15877.169988] Lustre: server umount lustre-MDT0001 complete [15905.097847] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing set_hostid [15915.542743] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing load_modules_local [15947.790694] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15702 sec ========= 12:55:59 (1787590559) [15949.373659] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:56:01 (1787590561) === [15952.554576] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:56:04 (1787590564) === [15991.888914] Lustre: DEBUG MARKER: oleg612-server.virtnet: executing unload_modules_local [15994.817604] Key type lgssc unregistered [15995.087845] LNet: 531820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15995.099124] LNetError: 531820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15995.126329] LNet: Removed LNI 192.168.206.112@tcp [15995.858732] Key type .llcrypt unregistered [15995.860926] Key type ._llcrypt unregistered