[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 511137721 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002381] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008012] pid_max: default: 32768 minimum: 301 [ 0.010112] LSM: Security Framework initializing [ 0.011060] Yama: becoming mindful. [ 0.012037] SELinux: Initializing. [ 0.013088] *** VALIDATE selinux *** [ 0.020792] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026268] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027144] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028105] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029103] *** VALIDATE tmpfs *** [ 0.030420] *** VALIDATE proc *** [ 0.031227] *** VALIDATE cgroup *** [ 0.032009] *** VALIDATE cgroup2 *** [ 0.034262] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035142] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037027] Spectre V2 : User space: Vulnerable [ 0.038009] Speculative Store Bypass: Vulnerable [ 0.041167] debug: unmapping init [mem 0xffffffffb6459000-0xffffffffb6460fff] [ 0.043918] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044641] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045017] ... version: 2 [ 0.046009] ... bit width: 48 [ 0.047037] ... generic registers: 4 [ 0.048008] ... value mask: 0000ffffffffffff [ 0.049010] ... max period: 00007fffffffffff [ 0.050012] ... fixed-purpose events: 3 [ 0.051010] ... event mask: 000000070000000f [ 0.052290] rcu: Hierarchical SRCU implementation. [ 0.054405] smp: Bringing up secondary CPUs ... [ 0.055416] x86: Booting SMP configuration: [ 0.056022] .... node #0, CPUs: #1 #2 #3 [ 0.060068] smp: Brought up 1 node, 4 CPUs [ 0.062017] smpboot: Max logical packages: 1 [ 0.063015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.132072] node 0 deferred pages initialised in 65ms [ 0.134102] devtmpfs: initialized [ 0.135200] x86/mm: Memory block size: 128MB [ 0.137662] gcov: version magic: 0x41383552 [ 0.139123] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.140088] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.141258] pinctrl core: initialized pinctrl subsystem [ 0.142164] [ 0.142574] ************************************************************* [ 0.143010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.144009] ** ** [ 0.145011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.146008] ** ** [ 0.147010] ** This means that this kernel is built to expose internal ** [ 0.148009] ** IOMMU data structures, which may compromise security on ** [ 0.149008] ** your system. ** [ 0.150010] ** ** [ 0.151009] ** If you see this message and you are not debugging the ** [ 0.152008] ** kernel, report this immediately to your vendor! ** [ 0.153009] ** ** [ 0.154010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.155008] ************************************************************* [ 0.156585] NET: Registered protocol family 16 [ 0.157405] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.158058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.159052] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.160388] cpuidle: using governor menu [ 0.161774] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.164542] PCI: Using configuration type 1 for base access [ 0.166137] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.176026] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.179017] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.183040] cryptd: max_cpu_qlen set to 1000 [ 0.185297] ACPI: Added _OSI(Module Device) [ 0.187011] ACPI: Added _OSI(Processor Device) [ 0.188010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.190013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.194745] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.202529] ACPI: Interpreter enabled [ 0.204046] ACPI: PM: (supports S0 S3 S4 S5) [ 0.206016] ACPI: Using IOAPIC for interrupt routing [ 0.207089] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.210436] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.220663] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.223037] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.225015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.228071] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.233082] acpiphp: Slot [2] registered [ 0.234127] acpiphp: Slot [5] registered [ 0.236101] acpiphp: Slot [6] registered [ 0.237064] acpiphp: Slot [7] registered [ 0.238323] acpiphp: Slot [8] registered [ 0.240112] acpiphp: Slot [9] registered [ 0.241116] acpiphp: Slot [10] registered [ 0.243102] acpiphp: Slot [3] registered [ 0.244125] acpiphp: Slot [4] registered [ 0.245084] acpiphp: Slot [11] registered [ 0.246081] acpiphp: Slot [12] registered [ 0.248106] acpiphp: Slot [13] registered [ 0.249107] acpiphp: Slot [14] registered [ 0.251091] acpiphp: Slot [15] registered [ 0.252056] acpiphp: Slot [16] registered [ 0.253083] acpiphp: Slot [17] registered [ 0.255070] acpiphp: Slot [18] registered [ 0.256121] acpiphp: Slot [19] registered [ 0.258094] acpiphp: Slot [20] registered [ 0.259104] acpiphp: Slot [21] registered [ 0.261100] acpiphp: Slot [22] registered [ 0.262108] acpiphp: Slot [23] registered [ 0.265123] acpiphp: Slot [24] registered [ 0.266084] acpiphp: Slot [25] registered [ 0.268099] acpiphp: Slot [26] registered [ 0.269090] acpiphp: Slot [27] registered [ 0.271075] acpiphp: Slot [28] registered [ 0.272105] acpiphp: Slot [29] registered [ 0.273071] acpiphp: Slot [30] registered [ 0.274127] acpiphp: Slot [31] registered [ 0.275046] PCI host bridge to bus 0000:00 [ 0.276012] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.278015] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.280016] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.282018] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.283018] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.286021] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.288053] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.291256] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.295104] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.304800] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.310548] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.313015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.315030] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.318021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.321530] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.323795] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.326034] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.328757] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.333012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.345014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.350013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.355386] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.363015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.372014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.390014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.398234] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.405014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.411013] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.427012] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.436340] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.449014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.457017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.479016] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.491808] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.500016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.506016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.524014] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.532047] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.538000] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.542019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.555015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.565415] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.573016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.582013] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.601000] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.610000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.612331] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.615373] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.617428] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.620201] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.624193] iommu: Default domain type: Passthrough [ 0.627366] SCSI subsystem initialized [ 0.628142] ACPI: bus type USB registered [ 0.630095] usbcore: registered new interface driver usbfs [ 0.631067] usbcore: registered new interface driver hub [ 0.633078] usbcore: registered new device driver usb [ 0.634095] pps_core: LinuxPPS API ver. 1 registered [ 0.635009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.638054] PTP clock support registered [ 0.640116] EDAC MC: Ver: 3.0.0 [ 0.642224] PCI: Using ACPI for IRQ routing [ 0.643550] NetLabel: Initializing [ 0.645009] NetLabel: domain hash size = 128 [ 0.646041] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.647071] NetLabel: unlabeled traffic allowed by default [ 0.650135] vgaarb: loaded [ 0.652370] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.653017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.657204] clocksource: Switched to clocksource kvm-clock [ 0.770479] VFS: Disk quotas dquot_6.6.0 [ 0.774225] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.777198] *** VALIDATE ramfs *** [ 0.778690] *** VALIDATE hugetlbfs *** [ 0.782398] pnp: PnP ACPI init [ 0.785312] pnp: PnP ACPI: found 6 devices [ 0.808883] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.813240] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.815626] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.818041] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.820452] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.822358] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.824550] NET: Registered protocol family 2 [ 0.826948] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.831840] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.835402] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.840104] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.844946] TCP: Hash tables configured (established 65536 bind 65536) [ 0.848280] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.851459] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.853889] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.857063] NET: Registered protocol family 1 [ 0.859566] RPC: Registered named UNIX socket transport module. [ 0.862161] RPC: Registered udp transport module. [ 0.864337] RPC: Registered tcp transport module. [ 0.866345] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.869213] NET: Registered protocol family 44 [ 0.871600] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.873754] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.875887] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.878512] PCI: CLS 0 bytes, default 64 [ 0.880147] Unpacking initramfs... [ 2.334720] debug: unmapping init [mem 0xffff97627cc54000-0xffff97627ffbffff] [ 2.338480] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.340516] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.342903] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.827503] Initialise system trusted keyrings [ 2.829377] Key type blacklist registered [ 2.831108] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.839393] zbud: loaded [ 2.842398] *** VALIDATE nfs *** [ 2.843383] *** VALIDATE nfs4 *** [ 2.844404] pstore: using deflate compression [ 2.847956] Platform Keyring initialized [ 2.966525] NET: Registered protocol family 38 [ 2.967840] Key type asymmetric registered [ 2.969062] Asymmetric key parser 'x509' registered [ 2.970671] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.973140] io scheduler mq-deadline registered [ 2.974703] io scheduler kyber registered [ 2.976072] io scheduler bfq registered [ 2.978204] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.981317] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.984155] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.986248] ACPI: Power Button [PWRF] [ 2.990971] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.997776] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.009481] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.015410] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.028104] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.058087] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.087489] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.092658] Non-volatile memory driver v1.3 [ 3.093926] Linux agpgart interface v0.103 [ 3.121651] virtio_blk virtio1: [vda] 145896 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.124105] vda: detected capacity change from 0 to 74698752 [ 3.142790] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.146778] vdb: detected capacity change from 0 to 1073741824 [ 3.172235] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.175598] vdc: detected capacity change from 0 to 2621440000 [ 3.199526] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.202851] vdd: detected capacity change from 0 to 2621440000 [ 3.225243] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.228594] vde: detected capacity change from 0 to 4294967296 [ 3.251312] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.254571] vdf: detected capacity change from 0 to 4294967296 [ 3.267374] libphy: Fixed MDIO Bus: probed [ 3.272900] usbcore: registered new interface driver usbserial_generic [ 3.275880] usbserial: USB Serial support registered for generic [ 3.278399] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.281529] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.283904] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.286904] mousedev: PS/2 mouse device common for all mice [ 3.292138] rtc_cmos 00:05: RTC can wake from S4 [ 3.294903] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.299652] rtc_cmos 00:05: registered as rtc0 [ 3.300575] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.301603] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.305749] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.307401] intel_pstate: CPU model not supported [ 3.313827] hid: raw HID events driver (C) Jiri Kosina [ 3.316511] usbcore: registered new interface driver usbhid [ 3.319046] usbhid: USB HID core driver [ 3.321698] drop_monitor: Initializing network drop monitor service [ 3.323906] Initializing XFRM netlink socket [ 3.325907] NET: Registered protocol family 10 [ 3.328655] Segment Routing with IPv6 [ 3.330325] NET: Registered protocol family 17 [ 3.332546] mpls_gso: MPLS GSO support [ 3.338838] RAS: Correctable Errors collector initialized. [ 3.340904] AVX version of gcm_enc/dec engaged. [ 3.342533] AES CTR mode by8 optimization enabled [ 3.419947] sched_clock: Marking stable (3419928713, 0)->(4334087338, -914158625) [ 3.423762] registered taskstats version 1 [ 3.426394] Loading compiled-in X.509 certificates [ 3.428624] zswap: loaded using pool lzo/zbud [ 3.448953] Key type big_key registered [ 3.458964] Key type encrypted registered [ 3.460372] ima: No TPM chip found, activating TPM-bypass! [ 3.462339] ima: Allocated hash algorithm: sha1 [ 3.463760] ima: No architecture policies found [ 3.465173] evm: Initialising EVM extended attributes: [ 3.466781] evm: security.selinux [ 3.468223] evm: security.ima [ 3.469485] evm: security.capability [ 3.470825] evm: HMAC attrs: 0x1 [ 3.473799] rtc_cmos 00:05: setting system clock to 2026-09-03 13:13:34 UTC (1788441214) [ 3.480169] debug: unmapping init [mem 0xffffffffb7403000-0xffffffffb75fffff] [ 3.483580] debug: unmapping init [mem 0xffffffffb6182000-0xffffffffb6458fff] [ 3.495637] Write protecting the kernel read-only data: 28672k [ 3.498338] debug: unmapping init [mem 0xffffffffb4803000-0xffffffffb49fffff] [ 3.501123] debug: unmapping init [mem 0xffffffffb5114000-0xffffffffb51fffff] [ 3.533401] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.542135] systemd[1]: Detected virtualization kvm. [ 3.544091] systemd[1]: Detected architecture x86-64. [ 3.545818] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.574216] systemd[1]: No hostname configured. [ 3.575830] systemd[1]: Set hostname to . [ 3.577852] random: systemd: uninitialized urandom read (16 bytes read) [ 3.580123] systemd[1]: Initializing machine ID from random generator. [ 3.617624] random: ln: uninitialized urandom read (6 bytes read) [ 3.705186] random: systemd: uninitialized urandom read (16 bytes read) [ 3.707831] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.714550] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.718197] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.265280] device-mapper: uevent: version 1.0.3 [ 4.267635] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.924440] random: fast init done [ 4.941386] virtio_net virtio0 ens2: renamed from eth0 [ 4.982888] scsi host0: ata_piix [ 5.018812] scsi host1: ata_piix [ 5.020251] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.022176] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.210227] dracut-initqueue[588]: RTNETLINK answers: File exists [ 9.976742] random: crng init done [ 9.978434] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.487209] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 13.189438] printk: systemd: 26 output lines suppressed due to ratelimiting [ 13.935702] SELinux: Disabled at runtime. [ 14.062718] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 14.075704] systemd[1]: Detected virtualization kvm. [ 14.081203] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 15.526835] systemd[1]: initrd-switch-root.service: Succeeded. [ 15.530566] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 15.538797] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 15.544837] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 15.550898] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 15.562821] systemd[1]: Starting Journal Service... Starting Journal Service... [ 15.575878] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ 15.851918] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Slices. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 16.750464] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 17.392897] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 17.466269] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 17.853145] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 17.927889] EDAC sbridge: Ver: 1.1.2 [ 21.708739] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit)[ 22.527206] NFS: Registering the id_resolver key type [ 22.529686] Key type id_resolver registered [ 22.531325] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg655-server login: [ 79.708072] libcfs: loading out-of-tree module taints kernel. [ 79.764817] Key type ._llcrypt registered [ 79.772141] Key type .llcrypt registered [ 79.897927] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 98.181463] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 99.990550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 100.002732] alg: No test for adler32 (adler32-zlib) [ 101.668566] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 102.730334] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 104.528153] Key type lgssc registered [ 106.227077] Lustre: Echo OBD driver; http://www.lustre.org/ [ 123.317530] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 160.878928] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 173.046519] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 173.086508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 174.337257] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 174.364958] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 174.444046] Lustre: lustre-MDT0000: new disk, initializing [ 174.536354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 174.590736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 179.867715] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 188.714009] hrtimer: interrupt took 4947391 ns [ 194.054765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 194.170533] Lustre: 6512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 194.203576] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 194.210908] Lustre: Skipped 1 previous similar message [ 194.356671] Lustre: lustre-MDT0001: new disk, initializing [ 194.436128] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 194.477566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 194.488255] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 198.573427] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 203.395848] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 212.999876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 213.318933] Lustre: lustre-OST0000: new disk, initializing [ 213.321943] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 213.328049] Lustre: 8450:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 213.393320] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 220.153906] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 222.770243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 222.786140] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 222.866807] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 234.709734] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 234.871444] Lustre: lustre-OST0001: new disk, initializing [ 234.879916] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 234.885373] Lustre: 9523:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 234.958582] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 240.867055] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 243.237573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 243.259118] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 243.332583] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 252.381842] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 258.005240] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 263.720741] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing check_logdir /tmp/testlogs/ [ 268.152419] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing yml_node [ 272.638810] Lustre: DEBUG MARKER: Client: 2.17.56.51 [ 275.168927] Lustre: DEBUG MARKER: MDS: 2.17.56.51 [ 277.724274] Lustre: DEBUG MARKER: OSS: 2.17.56.51 [ 279.240110] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 09:18:09 EDT 2026 [ 295.129568] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 296.425720] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 312.289601] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 312.293479] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 312.299700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 314.849509] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 314.852108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 314.870334] Lustre: Skipped 1 previous similar message [ 314.885429] Lustre: Skipped 3 previous similar messages [ 316.976493] Lustre: server umount lustre-MDT0000 complete [ 320.355946] LustreError: 6502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441531 with bad export cookie 562640900040872764 [ 320.363610] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 320.365768] LustreError: 6502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 325.091223] LustreError: 6518:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 325.091786] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 325.101834] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 325.103328] LustreError: 6518:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 325.124127] Lustre: Skipped 2 previous similar messages [ 326.852712] Lustre: server umount lustre-MDT0001 complete [ 330.707586] Lustre: server umount lustre-OST0000 complete [ 334.920682] Lustre: server umount lustre-OST0001 complete [ 341.720183] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 350.937387] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 384.650567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 384.919503] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 384.977632] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 385.065922] Lustre: lustre-MDT0000: new disk, initializing [ 385.152211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 385.174091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 389.706788] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 400.697071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 400.822040] Lustre: 15070:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 400.907292] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 400.911439] Lustre: Skipped 1 previous similar message [ 401.005870] Lustre: lustre-MDT0001: new disk, initializing [ 401.128509] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 401.166920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 401.176681] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 407.025201] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 416.330700] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 421.936349] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 429.426076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 429.718161] Lustre: lustre-OST0000: new disk, initializing [ 429.721736] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 429.727380] Lustre: 17043:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 429.811156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 431.219090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 431.230421] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 431.396272] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 436.401883] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 446.151643] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 450.933615] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 451.175433] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 454.743783] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 454.914115] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 457.185940] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 457.191413] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 457.200606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 457.200614] Lustre: Skipped 1 previous similar message [ 457.234080] Lustre: Skipped 1 previous similar message [ 462.312250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 462.314953] Lustre: Skipped 2 previous similar messages [ 462.603927] Lustre: server umount lustre-OST0000 complete [ 467.433868] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 472.548171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 472.559721] Lustre: Skipped 3 previous similar messages [ 472.680118] Lustre: server umount lustre-MDT0000 complete [ 477.403791] LustreError: 15062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441688 with bad export cookie 562640900040874297 [ 477.408984] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 477.420561] LustreError: 15062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 477.717294] Lustre: server umount lustre-MDT0001 complete [ 483.119111] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 09:21:32 (1788441692) [ 485.619463] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 487.491337] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 09:21:37 (1788441697) [ 493.752994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 494.045690] LustreError: 19451:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 494.098824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 498.786888] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 499.173654] LustreError: 19452:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 504.292673] LustreError: 19451:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 507.095243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 511.717664] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 519.274683] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 525.208915] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 531.629389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 537.944816] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 546.003204] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 554.470151] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 554.473798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 554.490581] Lustre: Skipped 2 previous similar messages [ 554.509329] Lustre: Skipped 1 previous similar message [ 560.047689] Lustre: server umount lustre-OST0000 complete [ 570.063506] Lustre: server umount lustre-MDT0000 complete [ 573.256692] LustreError: 19434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441784 with bad export cookie 562640900040875179 [ 573.265781] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 573.269864] LustreError: 19434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 573.635544] Lustre: server umount lustre-MDT0001 complete [ 583.835217] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 586.611856] Key type lgssc unregistered [ 586.878593] LNet: 22985:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 586.887164] LNetError: 22985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 586.899251] LNet: Removed LNI 192.168.206.155@tcp [ 587.756159] Key type .llcrypt unregistered [ 587.759256] Key type ._llcrypt unregistered [ 607.631954] Key type ._llcrypt registered [ 607.634923] Key type .llcrypt registered [ 607.717507] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 608.455416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 608.481055] alg: No test for adler32 (adler32-zlib) [ 609.497359] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 609.712712] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 611.352149] Key type lgssc registered [ 612.348492] Lustre: Echo OBD driver; http://www.lustre.org/ [ 618.789294] loop: module loaded [ 618.848409] loop0: detected capacity change from 0 to 204800000 [ 619.011524] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 619.022249] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 619.314699] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 623.800476] loop0: detected capacity change from 0 to 204800000 [ 623.891080] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 623.903649] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 623.911624] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 627.830923] loop0: detected capacity change from 0 to 204800000 [ 632.958334] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 633.017607] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 634.316958] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 634.418570] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 634.525294] Lustre: lustre-MDT0000: new disk, initializing [ 634.634267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 634.668309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 638.893320] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 646.994063] loop1: detected capacity change from 0 to 204800000 [ 652.963331] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 653.254685] Lustre: lustre-OST07c6: new disk, initializing [ 653.259494] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 653.269958] Lustre: Skipped 1 previous similar message [ 653.278820] Lustre: 26162:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 653.376970] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 659.011090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 659.020376] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 659.065688] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 659.770788] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 665.386698] Lustre: 25150:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 255 < left 264, rollback = 2 [ 665.393158] Lustre: 25150:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 665.397348] Lustre: 25150:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 665.401838] Lustre: 25150:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 665.406043] Lustre: 25150:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 665.410407] Lustre: 25150:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 681.410133] Lustre: 25150:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 681.417503] Lustre: 25150:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 681.422342] Lustre: 25150:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 681.427389] Lustre: 25150:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 681.432489] Lustre: 25150:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 681.437301] Lustre: 25150:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 686.029379] Lustre: 25150:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 686.039786] Lustre: 25150:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 686.045034] Lustre: 25150:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 686.050315] Lustre: 25150:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 686.054654] Lustre: 25150:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 686.061314] Lustre: 25150:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 686.067241] Lustre: 25150:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 686.073181] Lustre: 25150:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 686.077746] Lustre: 25150:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 686.083477] Lustre: 25150:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 686.089111] Lustre: 25150:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 686.093630] Lustre: 25150:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 689.253871] Lustre: 25150:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 689.263614] Lustre: 25150:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 689.268943] Lustre: 25150:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 689.273687] Lustre: 25150:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 689.280585] Lustre: 25150:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 689.290543] Lustre: 25150:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 689.296545] Lustre: 25150:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 689.303616] Lustre: 25150:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 689.312421] Lustre: 25150:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 689.317722] Lustre: 25150:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 689.327903] Lustre: 25150:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 689.340925] Lustre: 25150:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 693.827760] Lustre: 25151:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 693.846137] Lustre: 25151:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 693.854846] Lustre: 25151:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 693.860594] Lustre: 25151:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 693.865001] Lustre: 25151:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 693.870181] Lustre: 25151:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 693.874958] Lustre: 25151:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 693.880382] Lustre: 25151:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 693.885140] Lustre: 25151:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 693.889774] Lustre: 25151:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 693.897138] Lustre: 25151:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 693.903639] Lustre: 25151:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 703.531266] Lustre: 25149:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 703.542978] Lustre: 25149:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 703.549048] Lustre: 25149:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 703.557668] Lustre: 25149:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 703.578530] Lustre: 25149:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 703.586598] Lustre: 25149:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 703.593543] Lustre: 25149:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 703.607729] Lustre: 25149:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 703.616661] Lustre: 25149:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 703.625715] Lustre: 25149:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 703.629694] Lustre: 25149:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 703.640175] Lustre: 25149:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 756.195241] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 756.210107] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 756.236644] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 757.896128] Lustre: server umount lustre-OST07c6 complete [ 767.112347] Lustre: server umount lustre-MDT0000 complete [ 783.154579] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 785.773842] Key type lgssc unregistered [ 786.005871] LNet: 28722:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 786.015447] LNetError: 28722:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 786.034812] LNet: Removed LNI 192.168.206.155@tcp [ 786.783551] Key type .llcrypt unregistered [ 786.787993] Key type ._llcrypt unregistered [ 802.372958] Key type ._llcrypt registered [ 802.377667] Key type .llcrypt registered [ 802.515152] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 803.432095] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 803.503044] alg: No test for adler32 (adler32-zlib) [ 804.533993] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 804.758704] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 806.424283] Key type lgssc registered [ 807.528274] Lustre: Echo OBD driver; http://www.lustre.org/ [ 814.550911] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 814.580756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 816.053862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 820.301223] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 826.338105] LustreError: 29971:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 826.353388] LustreError: 29971:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 827.737132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 827.978513] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 831.306500] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 837.553489] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 842.111807] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 848.281475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 848.660719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 855.498823] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 861.730342] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 862.009684] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 877.538574] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 877.543243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 877.549871] Lustre: Skipped 1 previous similar message [ 881.439512] Lustre: server umount lustre-OST0000 complete [ 887.779198] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 887.780907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 887.800316] Lustre: Skipped 1 previous similar message [ 887.814546] Lustre: Skipped 2 previous similar messages [ 891.442887] Lustre: server umount lustre-MDT0000 complete [ 892.898126] LustreError: 29971:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 892.912951] LustreError: 29971:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 894.923282] LustreError: 29953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442105 with bad export cookie 1927352957792271341 [ 894.933551] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 895.329676] Lustre: server umount lustre-MDT0001 complete [ 904.461603] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 907.088934] Key type lgssc unregistered [ 907.334853] LNet: 33709:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 907.339748] LNetError: 33709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 907.350025] LNet: Removed LNI 192.168.206.155@tcp [ 908.143225] Key type .llcrypt unregistered [ 908.146288] Key type ._llcrypt unregistered [ 925.923898] Key type ._llcrypt registered [ 925.926724] Key type .llcrypt registered [ 926.051203] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 926.940275] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 926.959055] alg: No test for adler32 (adler32-zlib) [ 928.059413] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 928.288052] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 930.008262] Key type lgssc registered [ 931.162405] Lustre: Echo OBD driver; http://www.lustre.org/ [ 938.132858] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 938.158436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 939.650215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 944.657871] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 949.221983] LustreError: 34980:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 949.250134] LustreError: 34980:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 952.860642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 953.217345] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 957.274976] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 964.597964] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 969.011139] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 974.705043] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 975.150375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 980.752740] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 984.383477] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 987.681528] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 998.881419] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 998.887855] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 998.900116] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 999.909645] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 999.910953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 999.914545] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1004.131723] Lustre: server umount lustre-OST0000 complete [ 1009.633531] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1009.642265] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1009.651081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1009.655219] Lustre: Skipped 1 previous similar message [ 1013.488387] Lustre: server umount lustre-MDT0000 complete [ 1015.266850] LustreError: 34976:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1015.291723] LustreError: 34976:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1016.716216] LustreError: 34961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442227 with bad export cookie 17521964253576490151 [ 1016.729945] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1016.733707] LustreError: 34961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1017.059963] Lustre: server umount lustre-MDT0001 complete [ 1026.350764] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 1029.199277] Key type lgssc unregistered [ 1029.506930] LNet: 38515:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1029.513781] LNetError: 38515:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1029.532200] LNet: Removed LNI 192.168.206.155@tcp [ 1030.392537] Key type .llcrypt unregistered [ 1030.395078] Key type ._llcrypt unregistered [ 1050.563367] Key type ._llcrypt registered [ 1050.565236] Key type .llcrypt registered [ 1050.667481] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 1051.620682] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1051.699281] alg: No test for adler32 (adler32-zlib) [ 1052.813283] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1053.030466] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 1054.672150] Key type lgssc registered [ 1055.602229] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1062.327989] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1062.341575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1063.666864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1067.511583] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1073.125345] LustreError: 39912:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1073.148435] LustreError: 39912:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1074.595596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1074.950246] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1079.116288] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1085.900562] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1090.866449] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1096.444667] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1096.762421] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1102.614215] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1108.025601] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1109.902183] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1116.133878] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1116.145161] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1116.157623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1116.968838] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.55@tcp (stopping) [ 1117.584430] Lustre: server umount lustre-MDT0000 complete [ 1118.180589] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1118.190027] LustreError: 39912:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1118.201084] Lustre: Skipped 1 previous similar message [ 1118.219705] LustreError: 39912:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1120.857916] LustreError: 39894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442331 with bad export cookie 6211634922395430994 [ 1120.875754] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1120.876637] LustreError: 39894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1122.046960] LustreError: 39908:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1122.084930] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.55@tcp (stopping) [ 1123.297280] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1125.355399] LustreError: 39908:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1125.358349] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1125.371735] LustreError: 39908:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1125.376190] Lustre: Skipped 1 previous similar message [ 1127.273739] Lustre: server umount lustre-MDT0001 complete [ 1148.553892] Lustre: server umount lustre-OST0000 complete [ 1164.010552] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 1166.683828] Key type lgssc unregistered [ 1166.942627] LNet: 43775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1166.956416] LNetError: 43775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1166.977817] LNet: Removed LNI 192.168.206.155@tcp [ 1167.861697] Key type .llcrypt unregistered [ 1167.863073] Key type ._llcrypt unregistered [ 1190.560362] Key type ._llcrypt registered [ 1190.562987] Key type .llcrypt registered [ 1190.667318] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 1191.727514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1191.751719] alg: No test for adler32 (adler32-zlib) [ 1192.795849] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1193.129795] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 1194.792202] Key type lgssc registered [ 1195.953763] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1203.136934] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1203.157766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1204.691704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1208.917059] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1214.947701] LustreError: 45193:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1214.968163] LustreError: 45193:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1217.016097] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1217.418717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1221.936533] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1229.200729] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1235.176840] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1241.788951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1242.131658] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1248.457448] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1249.292696] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1255.407652] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1262.360895] Lustre: lustre-OST0000: Not available for connect from 192.168.206.55@tcp (stopping) [ 1262.562111] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1262.566708] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1264.617810] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1264.630573] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1264.633759] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1264.658358] Lustre: Skipped 2 previous similar messages [ 1266.833739] Lustre: server umount lustre-OST0000 complete [ 1284.068494] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1284.079301] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1284.093169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1289.186445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1289.198400] Lustre: Skipped 3 previous similar messages [ 1294.305780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1294.317904] Lustre: Skipped 1 previous similar message [ 1298.400865] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1298.742730] Lustre: server umount lustre-MDT0000 complete [ 1299.427946] LustreError: 46207:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1299.447068] LustreError: 46207:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1302.642981] LustreError: 45175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442513 with bad export cookie 9172882868854896056 [ 1302.655341] LustreError: 45175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1302.655408] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1309.025369] Lustre: server umount lustre-MDT0001 complete [ 1320.102273] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 1323.191673] Key type lgssc unregistered [ 1323.689424] LNet: 48906:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1323.705862] LNetError: 48906:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1323.731023] LNet: Removed LNI 192.168.206.155@tcp [ 1324.700186] Key type .llcrypt unregistered [ 1324.703416] Key type ._llcrypt unregistered [ 1348.959459] Key type ._llcrypt registered [ 1348.961698] Key type .llcrypt registered [ 1349.100880] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 1349.928405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1349.990944] alg: No test for adler32 (adler32-zlib) [ 1351.059918] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1351.314968] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 1352.992174] Key type lgssc registered [ 1354.128505] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1361.373535] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1361.404561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1362.827501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1367.414338] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1372.132163] LustreError: 50324:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1372.156682] LustreError: 50324:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1375.271449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1375.704892] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1379.825739] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1386.709207] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1391.428400] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1398.001666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1398.324276] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1405.079107] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1407.559249] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1411.878743] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1416.921375] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1420.558719] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1432.412665] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-03 9h37m22s [ 1433.958330] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1437.155156] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1437.163064] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1437.172764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1438.245320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1438.250889] Lustre: Skipped 4 previous similar messages [ 1440.063244] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.55@tcp (stopping) [ 1440.073196] Lustre: Skipped 1 previous similar message [ 1441.927617] Lustre: server umount lustre-MDT0000 complete [ 1444.331362] LustreError: 50319:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1444.373027] LustreError: 50319:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1445.490805] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1450.488609] LustreError: 52095:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1450.507302] LustreError: 52095:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1451.991752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1452.066488] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1452.235443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1452.280935] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1453.247161] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1457.113152] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1460.517531] Lustre: DEBUG MARKER: Wait for df (36733) ... [ 1461.821945] Lustre: DEBUG MARKER: done [ 1466.851056] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1466.856921] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1466.872849] Lustre: Skipped 2 previous similar messages [ 1466.882295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1467.877265] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1467.889520] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1474.023650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1474.034554] Lustre: Skipped 4 previous similar messages [ 1480.163030] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1480.292081] Lustre: server umount lustre-OST0000 complete [ 1489.390037] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1489.391263] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1489.419227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1489.419234] Lustre: Skipped 2 previous similar messages [ 1490.737658] Lustre: server umount lustre-MDT0000 complete [ 1494.143746] LustreError: 50305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442705 with bad export cookie 4428314685390888739 [ 1494.147772] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1494.154624] LustreError: 50305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1494.688318] Lustre: server umount lustre-MDT0001 complete [ 1503.331732] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 1505.852928] Key type lgssc unregistered [ 1506.139025] LNet: 55250:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1506.160216] LNetError: 55250:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1506.186711] LNet: Removed LNI 192.168.206.155@tcp [ 1506.961923] Key type .llcrypt unregistered [ 1506.963958] Key type ._llcrypt unregistered [ 1547.346449] Key type ._llcrypt registered [ 1547.350935] Key type .llcrypt registered [ 1547.511867] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 1548.678188] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1548.839014] alg: No test for adler32 (adler32-zlib) [ 1549.933882] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1550.138234] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 1551.875343] Key type lgssc registered [ 1552.915517] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1560.919667] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1560.952544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1562.266959] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1562.319291] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1562.501554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1567.172917] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1575.677110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1575.761630] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1575.808272] Lustre: 57460:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1576.138220] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1580.473964] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1587.894689] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1593.216313] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1599.678549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1599.905091] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1600.117844] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1606.316611] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1610.384206] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1613.766490] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1621.022326] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-03 9h40m31s [ 1622.525284] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1629.671784] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1632.470449] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1632.512108] LustreError: 57468:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1653.041628] Lustre: lustre-MDT0000: Client f21b5c0b-bc2f-4e1e-8eed-68faf52658b8 (at 192.168.206.55@tcp) reconnecting [ 1672.536778] LustreError: 57468:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1675.076148] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1677.731954] Lustre: DEBUG MARKER: done [ 1687.009971] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1687.019980] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1687.034552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1687.529079] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1687.541562] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1687.565444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1691.023952] Lustre: server umount lustre-OST0000 complete [ 1697.764949] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1697.770698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1697.797602] Lustre: Skipped 1 previous similar message [ 1697.811472] Lustre: Skipped 1 previous similar message [ 1701.160533] Lustre: server umount lustre-MDT0000 complete [ 1704.681479] LustreError: 57455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442915 with bad export cookie 4559827651656075983 [ 1704.689870] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1704.694694] LustreError: 57455:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1705.039561] Lustre: server umount lustre-MDT0001 complete [ 1715.445842] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 1717.896600] Key type lgssc unregistered [ 1718.228757] LNet: 61794:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1718.235932] LNetError: 61794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1718.250611] LNet: Removed LNI 192.168.206.155@tcp [ 1719.109229] Key type .llcrypt unregistered [ 1719.111837] Key type ._llcrypt unregistered [ 1758.826629] Key type ._llcrypt registered [ 1758.833658] Key type .llcrypt registered [ 1759.054739] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 1761.019871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1761.054426] alg: No test for adler32 (adler32-zlib) [ 1762.157789] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 1762.382201] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 1764.136168] Key type lgssc registered [ 1765.320234] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1768.738963] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1768.773316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1770.117762] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1770.157490] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1770.274097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1771.451601] Lustre: Failing over lustre-MDT0000 [ 1771.872606] Lustre: server umount lustre-MDT0000 complete [ 1778.768947] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 09:43:09 (1788442989) [ 1785.703216] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1786.063935] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1790.440763] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1797.786164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1797.909484] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1797.952516] Lustre: 64570:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1798.282530] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1802.705072] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1809.282634] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1814.244910] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1821.033117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1821.264772] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1821.492982] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1827.936899] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1829.539265] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1835.485668] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1841.654386] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1849.313031] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1849.327182] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1849.341986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1850.338792] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1850.342921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1850.350354] Lustre: Skipped 1 previous similar message [ 1850.357660] Lustre: Skipped 2 previous similar messages [ 1854.336776] Lustre: server umount lustre-MDT0000 complete [ 1857.976222] LustreError: 64565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443069 with bad export cookie 4115575222435114827 [ 1857.980093] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1857.985102] LustreError: 64565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1858.413235] Lustre: server umount lustre-MDT0001 complete [ 1861.851335] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1866.464590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1876.002233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1876.960128] Lustre: 63497:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788443071/real 1788443071] req@00000000c2b0771b x1875318378896384/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788443087 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1877.004419] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1877.986800] Lustre: 63496:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788443073/real 1788443073] req@0000000075bee770 x1875318378896512/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788443089 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1881.056188] Lustre: 63498:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788443076/real 1788443076] req@00000000402f1e52 x1875318378896640/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788443092 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1883.573114] LustreError: 66421:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1883.594441] LustreError: 66421:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1883.710357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1883.827356] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1884.704084] Lustre: 63499:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788443079/real 1788443079] req@00000000622b0428 x1875318378896896/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788443095 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1885.730286] LustreError: 66422:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1887.626620] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1887.794494] LustreError: 68344:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1887.820167] LustreError: 68344:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1889.888929] Lustre: 63496:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788443084/real 1788443084] req@00000000e885bb6e x1875318378897024/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788443100 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1892.838387] LustreError: 66421:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1892.860979] LustreError: 66421:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1896.329798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1896.674375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1902.058419] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1902.068227] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1905.201049] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1910.422554] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1916.566555] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1925.603976] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1925.614280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1925.627372] Lustre: Skipped 2 previous similar messages [ 1925.637346] Lustre: Skipped 2 previous similar messages [ 1928.930607] Lustre: server umount lustre-MDT0000 complete [ 1930.743974] LustreError: 68343:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1930.751484] LustreError: 68343:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1932.264748] LustreError: 64563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443143 with bad export cookie 4115575222435117102 [ 1932.277405] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1932.283082] LustreError: 64563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1932.700150] Lustre: server umount lustre-MDT0001 complete [ 1939.317186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1947.800669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1951.200460] Lustre: 63499:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788443146/real 1788443146] req@00000000f6836d3e x1875318378924544/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788443162 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1951.221167] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1957.815511] LustreError: 66422:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1957.897458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1957.968724] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1962.450875] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1970.704759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1975.463253] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1976.301966] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1980.403636] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1982.736617] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1987.753250] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1994.433552] Lustre: server umount lustre-MDT0000 complete [ 1995.751281] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1995.767188] LustreError: 71024:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1995.778853] LustreError: 71024:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 1997.675126] LustreError: 64565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443208 with bad export cookie 4115575222435118481 [ 1997.688282] LustreError: 64565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1997.690467] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2000.147860] Lustre: server umount lustre-MDT0001 complete [ 2006.336678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2009.594285] Lustre: DEBUG MARKER: files compared the same [ 2021.875201] Lustre: server umount lustre-OST0000 complete [ 2036.227771] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 2038.936826] Key type lgssc unregistered [ 2039.187627] LNet: 74303:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2039.194159] LNetError: 74303:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2039.233470] LNet: Removed LNI 192.168.206.155@tcp [ 2039.893768] Key type .llcrypt unregistered [ 2039.895405] Key type ._llcrypt unregistered [ 2060.029311] Key type ._llcrypt registered [ 2060.036181] Key type .llcrypt registered [ 2060.201182] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 2061.267799] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2061.396599] alg: No test for adler32 (adler32-zlib) [ 2062.437331] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2062.730160] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 2064.392263] Key type lgssc registered [ 2065.665265] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2073.429346] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2073.445392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2074.946098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2079.380326] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug +malloc all [ 2084.322328] LustreError: 75701:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2084.333258] LustreError: 75701:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2087.047450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2087.271428] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2091.139183] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug +malloc all [ 2098.550739] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2103.527255] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2109.464401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2109.703204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2115.235962] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug +malloc all [ 2119.990199] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2122.491337] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2130.400695] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2130.401750] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2130.410171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2130.423770] Lustre: Skipped 1 previous similar message [ 2130.441317] Lustre: Skipped 1 previous similar message [ 2135.522564] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2135.526725] Lustre: Skipped 1 previous similar message [ 2135.790255] Lustre: server umount lustre-OST0000 complete [ 2140.652541] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2140.661408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2140.680517] Lustre: Skipped 1 previous similar message [ 2145.762128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2145.772604] Lustre: Skipped 2 previous similar messages [ 2145.843749] Lustre: server umount lustre-MDT0000 complete [ 2149.381685] LustreError: 75682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443360 with bad export cookie 9575115988466054693 [ 2149.385546] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2149.392297] LustreError: 75682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2149.940648] Lustre: server umount lustre-MDT0001 complete [ 2160.489307] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 2163.401704] Key type lgssc unregistered [ 2163.662616] LNet: 79235:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2163.672495] LNetError: 79235:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2163.692813] LNet: Removed LNI 192.168.206.155@tcp [ 2164.500807] Key type .llcrypt unregistered [ 2164.504745] Key type ._llcrypt unregistered [ 2186.966609] Key type ._llcrypt registered [ 2186.969299] Key type .llcrypt registered [ 2187.096740] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 2187.832520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2187.942873] alg: No test for adler32 (adler32-zlib) [ 2189.272977] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2189.525505] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 2191.216273] Key type lgssc registered [ 2192.508673] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2199.435240] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2199.455581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2215.074048] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2220.193435] LustreError: 80626:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.155@tcp: failed processing log, type 4: rc = -110 [ 2246.880804] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2252.729818] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2260.517213] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2267.712071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2267.820547] Lustre: *** cfs_fail_loc=706, val=0*** [ 2268.185513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2268.243899] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2272.288845] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2273.271707] LustreError: 80651:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2273.286556] LustreError: 80651:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2278.375056] LustreError: 80652:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2279.790310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2280.106309] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2284.201557] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2290.373709] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2295.661843] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2298.850753] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2298.862129] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2298.881671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2300.902116] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2300.910429] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2300.928928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2301.932754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2303.616851] Lustre: server umount lustre-OST0000 complete [ 2311.143677] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2311.151991] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2311.166396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2311.170376] Lustre: Skipped 1 previous similar message [ 2313.512913] Lustre: server umount lustre-MDT0000 complete [ 2316.994427] LustreError: 80632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443528 with bad export cookie 16824979491342394683 [ 2316.996167] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2317.010156] LustreError: 80632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2317.379098] Lustre: server umount lustre-MDT0001 complete [ 2328.632305] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 2331.904018] Key type lgssc unregistered [ 2332.219899] LNet: 84247:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2332.227337] LNetError: 84247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2332.257160] LNet: Removed LNI 192.168.206.155@tcp [ 2333.118463] Key type .llcrypt unregistered [ 2333.120397] Key type ._llcrypt unregistered [ 2358.030785] Key type ._llcrypt registered [ 2358.034033] Key type .llcrypt registered [ 2358.170586] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 2359.043767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2359.078642] alg: No test for adler32 (adler32-zlib) [ 2360.393765] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2360.636901] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 2362.299557] Key type lgssc registered [ 2364.637801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2372.837992] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2372.855294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2378.486903] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2386.583329] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2387.282348] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2392.426858] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2392.551463] LustreError: 86356:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2392.578680] LustreError: 86356:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2397.673188] LustreError: 86360:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2397.690242] LustreError: 86360:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2399.998039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2400.244395] LustreError: 86356:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2400.250449] LustreError: 86356:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2400.266118] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2405.353390] LustreError: 87102:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2405.375349] LustreError: 87102:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2406.144788] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2410.479552] LustreError: 86359:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2410.504119] LustreError: 86359:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2414.916846] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2414.944706] Lustre: MGS: Client defdb516-0d3a-4b90-9b3f-b9329e7c4a9d (at 0@lo) reconnecting [ 2414.982299] LustreError: 86370:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2415.423028] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2415.571791] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2421.073449] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2436.579932] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2436.590240] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2436.619577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2441.186866] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2441.191522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2441.195617] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2441.232124] Lustre: Skipped 1 previous similar message [ 2442.335226] Lustre: server umount lustre-OST0000 complete [ 2446.305796] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2446.333741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2446.352431] Lustre: Skipped 1 previous similar message [ 2451.425996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2451.430108] Lustre: Skipped 1 previous similar message [ 2452.346619] Lustre: server umount lustre-MDT0000 complete [ 2456.278844] Lustre: server umount lustre-MDT0001 complete [ 2459.703572] Lustre: server umount MGS complete [ 2473.459688] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 09:54:43 (1788443683) [ 2502.718689] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 2512.520031] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 2543.657416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2548.099979] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2557.532216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2557.924304] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2557.975561] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2558.056750] Lustre: lustre-MDT0001: new disk, initializing [ 2558.126952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2558.153624] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2558.165508] Lustre: 93542:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2558.177338] LustreError: 93542:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2562.552346] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2572.765343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2573.118724] Lustre: lustre-OST0000: new disk, initializing [ 2573.131441] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2573.146372] Lustre: 94309:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2573.257701] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2579.118275] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2584.167420] Lustre: 94827:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000001a1bedb9 x1875319005945344/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788443811 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2588.019176] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2593.933310] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2593.946742] Lustre: MGS: Client 3977387a-9fea-4726-88db-e59459af0220 (at 0@lo) reconnecting [ 2594.001629] Lustre: 95355:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2594.020840] LustreError: 93528:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2594.058919] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2594.137589] Lustre: lustre-MDT0000: new disk, initializing [ 2594.273681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2594.302204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2594.407895] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2595.366369] Lustre: 95379:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000009b50510b x1875319005951872/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788443822 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2598.948455] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2604.454670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2604.468402] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2604.557531] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2629.096926] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2629.108606] Lustre: Skipped 2 previous similar messages [ 2629.110645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2629.126927] Lustre: Skipped 1 previous similar message [ 2639.333957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2639.339501] Lustre: Skipped 3 previous similar messages [ 2642.401544] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2642.550349] Lustre: server umount lustre-OST0000 complete [ 2649.578634] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2649.577175] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2649.610314] Lustre: Skipped 1 previous similar message [ 2659.814166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2659.819900] Lustre: Skipped 5 previous similar messages [ 2660.320125] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2660.431388] Lustre: server umount lustre-MDT0000 complete [ 2670.725664] Lustre: server umount lustre-MDT0001 complete [ 2674.490707] Lustre: server umount MGS complete [ 2684.852596] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 09:58:15 (1788443895) [ 2700.404529] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 2703.236864] Key type lgssc unregistered [ 2703.531576] LNet: 97885:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2703.550635] LNetError: 97885:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2703.566371] LNet: Removed LNI 192.168.206.155@tcp [ 2704.367685] Key type .llcrypt unregistered [ 2704.371912] Key type ._llcrypt unregistered [ 2719.318638] Key type ._llcrypt registered [ 2719.321310] Key type .llcrypt registered [ 2719.440222] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 2720.081082] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2720.130378] alg: No test for adler32 (adler32-zlib) [ 2721.197133] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2721.473042] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 2723.200161] Key type lgssc registered [ 2724.079432] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2731.012408] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2731.028723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2732.561748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2737.309905] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2742.760614] LustreError: 99085:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2742.802091] LustreError: 99085:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2745.754864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2746.460941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2750.927451] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2758.731730] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2764.579445] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2771.327542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2771.801380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2777.029090] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2780.917247] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2790.659405] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2798.561759] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2798.571906] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2798.587453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2802.657900] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2802.659640] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2802.671786] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2807.779627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2807.781672] Lustre: Skipped 1 previous similar message [ 2812.385853] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2812.619976] Lustre: server umount lustre-OST0000 complete [ 2818.016922] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2818.018776] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2818.030264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2818.040637] Lustre: Skipped 1 previous similar message [ 2818.050724] Lustre: Skipped 1 previous similar message [ 2822.496118] Lustre: server umount lustre-MDT0000 complete [ 2823.138916] LustreError: 99080:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2823.161697] LustreError: 99080:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2825.897559] LustreError: 99068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444036 with bad export cookie 970673822067879245 [ 2825.902574] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2825.907800] LustreError: 99068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2826.256121] Lustre: server umount lustre-MDT0001 complete [ 2838.832764] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 2841.719598] Key type lgssc unregistered [ 2842.106508] LNet: 102623:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2842.119786] LNetError: 102623:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2842.161294] LNet: Removed LNI 192.168.206.155@tcp [ 2843.245693] Key type .llcrypt unregistered [ 2843.248202] Key type ._llcrypt unregistered [ 2859.677834] Key type ._llcrypt registered [ 2859.681040] Key type .llcrypt registered [ 2859.804746] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 2860.480925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2860.547464] alg: No test for adler32 (adler32-zlib) [ 2861.652255] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 2861.882769] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 2863.608256] Key type lgssc registered [ 2864.631332] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2870.014202] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2870.014864] LustreError: 103725:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2870.017391] Lustre: Skipped 1 previous similar message [ 2870.017407] LustreError: 103724:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2870.029153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2870.033148] LustreError: 103725:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 2870.046096] LustreError: 103725:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2870.053774] LustreError: 103725:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2871.516120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2878.119879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2878.387680] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2882.159047] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2886.750435] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing lsmod [ 2889.960588] LustreError: 104914:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2890.026424] LustreError: 104931:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2890.031278] LustreError: 104914:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4935 [ 2890.039154] LustreError: 104914:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2890.044484] LustreError: 104914:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2890.051560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2890.338404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2893.794079] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2893.810900] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2893.829942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2895.330445] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2895.345840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2895.376519] Lustre: Skipped 1 previous similar message [ 2895.412597] Lustre: Skipped 2 previous similar messages [ 2900.449465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2903.523297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2903.529933] Lustre: Skipped 4 previous similar messages [ 2908.128148] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2908.431458] Lustre: server umount lustre-MDT0000 complete [ 2908.647216] LustreError: 103761:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2908.659301] LustreError: 103761:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2912.016945] LustreError: 103740:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444123 with bad export cookie 13600400599021913616 [ 2912.025674] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2913.762110] LustreError: 104192:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000bc77ec9d x1875319531263104/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2913.764072] LustreError: 104818:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2913.765128] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2913.768280] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2913.954118] Lustre: server umount lustre-MDT0001 complete [ 2919.724492] Lustre: server umount lustre-OST0000 complete [ 2927.587725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2928.083481] LustreError: 105993:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2928.152596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2932.152455] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2933.218382] LustreError: 105994:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2938.347376] LustreError: 105993:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2939.966730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2940.254387] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2944.348484] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2951.314693] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2956.479624] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2964.043525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2964.454614] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2970.474707] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2973.694600] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:65) [ 2976.629893] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2983.907153] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2983.909306] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2983.912394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2983.926468] Lustre: Skipped 1 previous similar message [ 2989.435856] Lustre: server umount lustre-OST0000 complete [ 2994.146510] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2999.265241] LustreError: 105990:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2999.286428] LustreError: 105990:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3000.968455] Lustre: server umount lustre-MDT0000 complete [ 3004.610788] LustreError: 105975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444215 with bad export cookie 13600400599021914316 [ 3004.631413] LustreError: 105975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3004.637416] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3004.925628] Lustre: server umount lustre-MDT0001 complete [ 3015.746205] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 3018.003209] Key type lgssc unregistered [ 3018.356380] LNet: 109525:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3018.371482] LNetError: 109525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3018.393184] LNet: Removed LNI 192.168.206.155@tcp [ 3019.285505] Key type .llcrypt unregistered [ 3019.287500] Key type ._llcrypt unregistered [ 3043.779379] Key type ._llcrypt registered [ 3043.782907] Key type .llcrypt registered [ 3043.881084] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 3044.734945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3044.795424] alg: No test for adler32 (adler32-zlib) [ 3045.785743] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3046.010371] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 3047.704140] Key type lgssc registered [ 3048.604717] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3055.158292] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3055.173499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3056.753540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3060.948965] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3066.338668] LustreError: 110940:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3066.345664] LustreError: 110940:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3069.191861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3069.471861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3074.942784] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3082.210664] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3087.333766] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3092.981061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3093.395252] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3098.965462] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3101.632607] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3106.138640] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3110.875807] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3114.846743] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3122.150305] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3122.154266] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3122.172442] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3122.208888] Lustre: Skipped 1 previous similar message [ 3126.880755] Lustre: server umount lustre-OST0000 complete [ 3130.850545] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3130.868430] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3130.900914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3130.913300] Lustre: Skipped 1 previous similar message [ 3132.385843] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3132.388224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3132.410407] Lustre: Skipped 1 previous similar message [ 3136.821701] Lustre: server umount lustre-MDT0000 complete [ 3137.505874] LustreError: 110935:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3137.530046] LustreError: 110935:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3139.945936] LustreError: 110923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444350 with bad export cookie 17027247168573614289 [ 3139.958036] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3139.974584] LustreError: 110923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3140.146264] Lustre: server umount lustre-MDT0001 complete [ 3149.408094] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 3152.474469] Key type lgssc unregistered [ 3152.695572] LNet: 114581:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3152.702150] LNetError: 114581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3152.714693] LNet: Removed LNI 192.168.206.155@tcp [ 3153.338800] Key type .llcrypt unregistered [ 3153.341367] Key type ._llcrypt unregistered [ 3171.218135] Key type ._llcrypt registered [ 3171.221646] Key type .llcrypt registered [ 3171.372930] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 3172.559350] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3172.618110] alg: No test for adler32 (adler32-zlib) [ 3173.711520] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3173.945205] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 3175.617374] Key type lgssc registered [ 3177.110342] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3184.877222] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3184.890310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3186.469659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3190.676276] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3196.899740] LustreError: 115810:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3196.921918] LustreError: 115810:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3197.552796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3197.770322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3202.494153] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3210.512205] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3215.966183] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3222.031177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3222.315104] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3228.209496] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3231.549595] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3235.210763] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3244.002055] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3244.011215] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3244.028657] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3245.036550] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3245.055214] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3245.068959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3247.090892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3249.329689] Lustre: server umount lustre-OST0000 complete [ 3254.241955] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3254.256508] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3254.283151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3254.289115] Lustre: Skipped 1 previous similar message [ 3257.314292] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3259.860181] Lustre: server umount lustre-MDT0000 complete [ 3262.433507] LustreError: 116813:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3262.472560] LustreError: 116813:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3263.374885] LustreError: 115792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444474 with bad export cookie 8553143533306644409 [ 3263.379694] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3263.383719] LustreError: 115792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3263.723817] Lustre: server umount lustre-MDT0001 complete [ 3274.095900] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 3276.978095] Key type lgssc unregistered [ 3277.265216] LNet: 119345:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3277.277850] LNetError: 119345:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3277.308318] LNet: Removed LNI 192.168.206.155@tcp [ 3278.189937] Key type .llcrypt unregistered [ 3278.196110] Key type ._llcrypt unregistered [ 3301.608658] Key type ._llcrypt registered [ 3301.611219] Key type .llcrypt registered [ 3301.723972] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 3302.953812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3303.012778] alg: No test for adler32 (adler32-zlib) [ 3304.180563] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3304.533342] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 3306.312150] Key type lgssc registered [ 3307.186940] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3314.801316] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3314.819300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3316.272627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3319.313493] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3326.436563] LustreError: 120761:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3326.444664] LustreError: 120761:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3326.538127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3326.912503] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3330.442915] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3336.558578] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3341.529983] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3347.580859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3347.851515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3352.750681] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3358.939811] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3361.198353] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 3371.412383] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3375.724580] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3375.746105] Lustre: Skipped 2 previous similar messages [ 3378.283552] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3378.330920] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3380.581437] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3380.586644] Lustre: Skipped 2 previous similar messages [ 3388.459385] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3388.462608] Lustre: Skipped 1 previous similar message [ 3392.156362] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.206.55@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3392.164490] Lustre: Skipped 2 previous similar messages [ 3398.242513] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.206.55@tcp 192.168.0.[2,10]@tcp [ 3404.468389] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.206.55/24@tcp in log lustre-MDT0000 [ 3404.474766] Lustre: Skipped 2 previous similar messages [ 3404.826897] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.55/24@tcp [ 3404.831246] Lustre: Skipped 1 previous similar message [ 3416.055853] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3416.058725] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3416.071974] LustreError: Skipped 2 previous similar messages [ 3416.131359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3417.573616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3417.576613] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3420.875317] Lustre: server umount lustre-OST0000 complete [ 3427.809205] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3427.811853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3427.820676] Lustre: Skipped 1 previous similar message [ 3427.826426] Lustre: Skipped 2 previous similar messages [ 3431.036448] Lustre: server umount lustre-MDT0000 complete [ 3432.933962] LustreError: 120761:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3432.943056] LustreError: 120761:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3434.074901] LustreError: 122604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444645 with bad export cookie 5936882519215212088 [ 3434.076654] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3434.083474] LustreError: 122604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3434.401203] Lustre: server umount lustre-MDT0001 complete [ 3443.015523] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 3445.338451] Key type lgssc unregistered [ 3445.607376] LNet: 125664:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3445.613740] LNetError: 125664:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3445.644622] LNet: Removed LNI 192.168.206.155@tcp [ 3446.410341] Key type .llcrypt unregistered [ 3446.412514] Key type ._llcrypt unregistered [ 3471.108633] Key type ._llcrypt registered [ 3471.111519] Key type .llcrypt registered [ 3471.217868] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 3472.251270] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3472.523737] alg: No test for adler32 (adler32-zlib) [ 3473.614655] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3473.848700] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 3475.568537] Key type lgssc registered [ 3476.843734] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3484.593040] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3484.619607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3485.964701] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3486.022084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3486.024550] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.206.55/24@tcp [ 3490.207818] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3495.396820] LustreError: 127225:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3495.418994] LustreError: 127225:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3498.850894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3499.306974] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3499.329424] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.55/24@tcp [ 3499.405176] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3503.049457] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3509.644670] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3515.522113] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3522.203203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3522.461405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3529.105466] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3530.686541] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:169 to 0x280000401:193) [ 3536.906450] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3550.177848] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3550.189285] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3550.208321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3551.205403] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3551.205903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3551.229340] Lustre: Skipped 1 previous similar message [ 3553.197553] Lustre: server umount lustre-OST0000 complete [ 3560.929638] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3560.936369] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3560.951955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3566.561896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3566.567777] Lustre: Skipped 3 previous similar messages [ 3571.168225] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3571.337258] Lustre: server umount lustre-MDT0000 complete [ 3571.681947] LustreError: 128315:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3571.712814] LustreError: 128315:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3574.839717] LustreError: 127206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444785 with bad export cookie 13244550905218032786 [ 3574.842587] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3574.847938] LustreError: 127206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3575.143713] Lustre: server umount lustre-MDT0001 complete [ 3584.252772] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 3586.988521] Key type lgssc unregistered [ 3587.305626] LNet: 130816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3587.317291] LNetError: 130816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3587.344914] LNet: Removed LNI 192.168.206.155@tcp [ 3588.338615] Key type .llcrypt unregistered [ 3588.340206] Key type ._llcrypt unregistered [ 3615.244396] Key type ._llcrypt registered [ 3615.246838] Key type .llcrypt registered [ 3615.375310] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 3627.969729] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 3628.684338] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3628.710893] alg: No test for adler32 (adler32-zlib) [ 3629.818988] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3630.064211] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 3631.753293] Key type lgssc registered [ 3632.879991] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3662.888612] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3662.931799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3664.268709] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3664.294746] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3664.377363] Lustre: lustre-MDT0000: new disk, initializing [ 3664.445597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3664.476469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3668.819496] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3678.612624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3678.815198] Lustre: 134198:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3678.900547] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3678.904688] Lustre: Skipped 1 previous similar message [ 3678.962689] Lustre: lustre-MDT0001: new disk, initializing [ 3679.053264] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3679.111097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3679.125231] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3683.428152] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3692.908698] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3699.028509] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3705.513156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3705.831456] Lustre: lustre-OST0000: new disk, initializing [ 3705.835306] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3705.841513] Lustre: 136166:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3705.934555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3711.576813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3711.588524] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3711.716068] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3712.676461] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3722.067564] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3726.364494] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3726.593972] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3731.153317] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3731.359493] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3737.575347] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3737.586943] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3737.601923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3737.607508] Lustre: Skipped 1 previous similar message [ 3739.451444] Lustre: server umount lustre-OST0000 complete [ 3745.761349] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3745.767132] LustreError: Skipped 1 previous similar message [ 3745.772303] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3745.778212] Lustre: Skipped 1 previous similar message [ 3745.781260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3747.809329] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3747.809366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3747.830529] Lustre: Skipped 1 previous similar message [ 3749.126520] Lustre: server umount lustre-MDT0000 complete [ 3752.245234] LustreError: 134189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444963 with bad export cookie 7514577311663826034 [ 3752.252754] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3752.262083] LustreError: 134189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3752.528550] Lustre: server umount lustre-MDT0001 complete [ 3759.179854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3759.637379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3763.020346] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3764.706991] LustreError: 138382:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3764.724111] LustreError: 138382:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3769.647091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3769.833706] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3769.922045] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3773.098162] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3779.570946] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3784.331525] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3789.957776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3790.202243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3794.887970] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3802.034480] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3807.369114] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3807.661878] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3813.467477] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3813.743149] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3823.487811] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3823.660046] Lustre: lustre-OST0001: new disk, initializing [ 3823.669828] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3823.684765] Lustre: 141754:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3823.802928] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3830.391723] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3832.462849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3832.474497] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3832.531536] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3838.529101] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3838.764965] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3843.210317] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3843.393391] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3848.533534] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8a0ac91e2000.ost_server_uuid 50 [ 3849.667875] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8a0ac91e2000.ost_server_uuid in IDLE state after 0 sec [ 3858.408265] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3858.409891] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3858.412061] LustreError: Skipped 1 previous similar message [ 3858.417477] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3858.425908] Lustre: Skipped 1 previous similar message [ 3862.935730] Lustre: server umount lustre-OST0001 complete [ 3863.540242] LustreError: 140204:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3863.556464] LustreError: 140204:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3866.081546] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3866.088696] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3866.111597] Lustre: Skipped 1 previous similar message [ 3868.646990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3868.650561] LustreError: 140202:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3868.656275] Lustre: Skipped 4 previous similar messages [ 3868.673306] LustreError: 140202:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3871.961574] Lustre: server umount lustre-OST0000 complete [ 3877.348319] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3877.357632] LustreError: Skipped 1 previous similar message [ 3877.363956] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3881.881753] Lustre: server umount lustre-MDT0000 complete [ 3884.002061] LustreError: 139670:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3884.013583] LustreError: 139670:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3885.745990] LustreError: 138363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445096 with bad export cookie 7514577311663826916 [ 3885.752224] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3885.763449] LustreError: 138363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3886.161377] Lustre: server umount lustre-MDT0001 complete [ 3906.141852] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 3908.652779] Key type lgssc unregistered [ 3908.894446] LNet: 144757:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3908.908427] LNetError: 144757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3909.934786] LNet: Removed LNI 192.168.206.155@tcp [ 3910.678506] Key type .llcrypt unregistered [ 3910.680591] Key type ._llcrypt unregistered [ 3946.429995] Key type ._llcrypt registered [ 3946.433084] Key type .llcrypt registered [ 3946.529576] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 3959.446278] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 3960.201882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3960.255083] alg: No test for adler32 (adler32-zlib) [ 3961.433915] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 3961.737040] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 3963.408734] Key type lgssc registered [ 3964.529759] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3996.468298] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3996.486619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3997.680751] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3997.711377] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3997.771282] Lustre: lustre-MDT0000: new disk, initializing [ 3997.842143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3997.896408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4002.199294] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4012.551633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4012.699726] Lustre: 148908:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4012.759333] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4012.762035] Lustre: Skipped 1 previous similar message [ 4012.811372] Lustre: lustre-MDT0001: new disk, initializing [ 4012.891362] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4012.930050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4012.935519] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4018.255786] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4029.698289] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4037.522062] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4045.395587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4045.914656] Lustre: lustre-OST0000: new disk, initializing [ 4045.929808] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4045.962588] Lustre: 150876:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4046.091127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4051.569981] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4052.502439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4052.508617] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4052.580331] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4058.980548] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4062.338472] Lustre: Failing over lustre-OST0000 [ 4062.537137] Lustre: server umount lustre-OST0000 complete [ 4079.073157] Lustre: 147129:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788445273/real 1788445273] req@0000000028a783d6 x1875320684707200/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788445289 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4079.093249] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4080.260382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4080.562659] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4080.582370] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4080.928222] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4081.633059] Lustre: 147130:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788445275/real 1788445275] req@000000003e296773 x1875320684707968/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788445291 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4081.656308] Lustre: 147130:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4083.608358] Lustre: 147128:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788445278/real 1788445278] req@000000002f4ac3b5 x1875320684708736/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788445294 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4086.209318] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4088.666388] Lustre: 147131:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788445283/real 1788445283] req@0000000071027c7f x1875320684710016/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788445299 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4088.698163] Lustre: 147131:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4089.654408] Lustre: Failing over lustre-MDT0000 [ 4089.668357] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4089.675780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4089.678421] Lustre: Skipped 3 previous similar messages [ 4089.688607] Lustre: Skipped 2 previous similar messages [ 4089.893449] Lustre: server umount lustre-MDT0000 complete [ 4094.817051] Lustre: 147130:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788445289/real 1788445289] req@00000000969c7d96 x1875320684710912/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788445305 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4094.856934] Lustre: 147130:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4095.970754] LustreError: 148915:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4095.985145] LustreError: 148915:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4101.093025] LustreError: 150076:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4101.123364] LustreError: 150076:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4106.209120] LustreError: 148919:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4106.225064] LustreError: 148919:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4106.693955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4106.806307] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4106.952672] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 4106.999199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4107.040407] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4110.490536] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4111.613178] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4112.365734] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4112.387044] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4112.400614] Lustre: Skipped 1 previous similar message [ 4112.440705] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4117.478679] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4117.479085] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4117.480931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4117.487692] Lustre: Skipped 1 previous similar message [ 4117.491987] LustreError: Skipped 1 previous similar message [ 4118.503437] Lustre: server umount lustre-OST0000 complete [ 4122.594295] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4122.594984] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4122.601578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4122.611566] Lustre: Skipped 1 previous similar message [ 4122.619226] Lustre: Skipped 2 previous similar messages [ 4127.714620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4127.720327] Lustre: Skipped 1 previous similar message [ 4132.833723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4132.840673] Lustre: Skipped 1 previous similar message [ 4135.392123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4135.572218] Lustre: server umount lustre-MDT0000 complete [ 4137.958508] LustreError: 150074:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4137.974486] LustreError: 150074:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4138.807593] LustreError: 148900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445349 with bad export cookie 9829820992248009886 [ 4138.821991] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4138.822229] LustreError: 148900:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4139.086729] Lustre: server umount lustre-MDT0001 complete [ 4149.362614] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 4151.940600] Key type lgssc unregistered [ 4152.116614] LNet: 154530:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4152.123714] LNetError: 154530:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4152.138540] LNet: Removed LNI 192.168.206.155@tcp [ 4152.706501] Key type .llcrypt unregistered [ 4152.709359] Key type ._llcrypt unregistered [ 4187.819478] Key type ._llcrypt registered [ 4187.821730] Key type .llcrypt registered [ 4187.921247] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 4188.897593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4188.931553] alg: No test for adler32 (adler32-zlib) [ 4189.949382] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 4190.201650] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 4191.928180] Key type lgssc registered [ 4193.327393] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4201.362283] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4201.418191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4203.171828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4208.515126] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4213.738310] LustreError: 156051:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4213.759395] LustreError: 156051:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4214.826586] LustreError: 156050:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4219.846402] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4219.879067] LustreError: 156051:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4223.986983] LustreError: 156079:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4224.000194] LustreError: 156079:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4224.020086] LustreError: 156079:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 4224.808320] Lustre: server umount lustre-MDT0000 complete [ 4240.255430] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 4250.147780] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 4287.560829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4287.780388] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4287.869845] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4287.965437] Lustre: lustre-MDT0000: new disk, initializing [ 4288.029344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4288.043640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4292.425376] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4303.006912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4303.184226] Lustre: 159383:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4303.277121] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4303.284663] Lustre: Skipped 1 previous similar message [ 4303.409200] Lustre: lustre-MDT0001: new disk, initializing [ 4303.546601] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4303.603168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4303.639905] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4308.265595] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4318.402453] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4324.606128] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4331.986800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4332.395997] Lustre: lustre-OST0000: new disk, initializing [ 4332.406028] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4332.424645] Lustre: 161351:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4332.593253] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4338.299586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4338.319120] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4338.456412] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4339.275948] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4350.248345] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4354.915448] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4356.737206] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 10:26:07 (1788445567) [ 4589.537027] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4589.537780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4589.544175] Lustre: Skipped 1 previous similar message [ 4594.661609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4594.667496] Lustre: Skipped 4 previous similar messages [ 4594.818905] Lustre: server umount lustre-MDT0000 complete [ 4597.988204] LustreError: 159375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445809 with bad export cookie 8833927386536643419 [ 4597.990432] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4598.001514] LustreError: 159375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4599.776703] LustreError: 161633:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4599.780742] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4599.808111] Lustre: Skipped 1 previous similar message [ 4599.810987] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4602.784882] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4604.395538] Lustre: server umount lustre-MDT0001 complete [ 4613.758124] Lustre: server umount lustre-OST0000 complete [ 4622.121363] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 4630.260533] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 4662.482562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4662.765366] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4662.795251] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4662.909147] Lustre: lustre-MDT0000: new disk, initializing [ 4662.996731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4663.010674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4666.911552] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4677.132100] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4677.280643] Lustre: 166006:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4677.354703] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4677.366970] Lustre: Skipped 1 previous similar message [ 4677.529784] Lustre: lustre-MDT0001: new disk, initializing [ 4677.675816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4677.717442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4677.724765] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4683.134870] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4694.204564] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4700.006192] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4706.311574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4706.598880] Lustre: lustre-OST0000: new disk, initializing [ 4706.608972] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4706.620043] Lustre: 167980:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4706.705620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4712.440694] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4713.557643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4713.582685] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4713.763470] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4721.474581] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4725.793038] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4725.964775] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4729.970750] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4730.160532] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4734.434742] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4734.437742] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4734.442279] LustreError: Skipped 1 previous similar message [ 4734.449862] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4734.453836] Lustre: Skipped 1 previous similar message [ 4744.712506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4744.716503] Lustre: Skipped 3 previous similar messages [ 4746.208161] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4746.578372] Lustre: server umount lustre-OST0000 complete [ 4754.405887] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4754.441108] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4754.489030] Lustre: Skipped 1 previous similar message [ 4757.211426] Lustre: server umount lustre-MDT0000 complete [ 4760.033942] LustreError: 166015:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4760.059345] LustreError: 166015:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4761.650498] LustreError: 165999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445972 with bad export cookie 8833927386536802249 [ 4761.659193] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4762.329314] Lustre: server umount lustre-MDT0001 complete [ 4771.329971] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:33:01 (1788445981) [ 4786.878243] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 4795.593186] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 4831.724561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4831.949234] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4831.977055] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4832.053483] Lustre: lustre-MDT0000: new disk, initializing [ 4832.112921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4832.126692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4835.571871] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4844.419550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4844.537623] Lustre: 172739:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4844.781595] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4849.074106] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4857.771437] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4862.884936] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4869.459253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4869.763459] Lustre: lustre-OST0000: new disk, initializing [ 4869.765654] Lustre: Skipped 1 previous similar message [ 4869.768844] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4869.772895] Lustre: Skipped 2 previous similar messages [ 4869.776965] Lustre: 174714:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4876.365073] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4876.382777] Lustre: Skipped 1 previous similar message [ 4876.397071] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4876.512317] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4876.516307] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4885.509904] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4892.131191] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4892.136178] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4892.137387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4892.137392] Lustre: Skipped 2 previous similar messages [ 4892.150083] Lustre: Skipped 2 previous similar messages [ 4892.160965] LustreError: Skipped 1 previous similar message [ 4893.920378] Lustre: server umount lustre-OST0000 complete [ 4903.499040] Lustre: server umount lustre-MDT0000 complete [ 4906.317127] LustreError: 172731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446117 with bad export cookie 8833927386536803131 [ 4906.318048] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4906.330094] LustreError: 172731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4906.564160] Lustre: server umount lustre-MDT0001 complete [ 4913.670268] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:35:24 (1788446124) [ 4926.955978] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 4935.427619] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 4969.374597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4969.601038] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4969.633245] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4969.718188] Lustre: lustre-MDT0000: new disk, initializing [ 4969.795600] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4969.802517] Lustre: Skipped 2 previous similar messages [ 4969.814115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4974.562527] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4984.279209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4984.400345] Lustre: 179031:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4984.666572] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4989.108654] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4997.617567] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5002.925187] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5010.629138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5010.962730] Lustre: 181011:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5018.165141] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5018.315851] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5018.374323] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5028.017571] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5039.076123] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5039.078237] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5039.094133] LustreError: Skipped 1 previous similar message [ 5039.132849] Lustre: Skipped 2 previous similar messages [ 5039.162296] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5039.176396] Lustre: Skipped 6 previous similar messages [ 5041.347267] Lustre: server umount lustre-OST0000 complete [ 5046.241660] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5051.031482] Lustre: server umount lustre-MDT0000 complete [ 5054.146944] LustreError: 179831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446265 with bad export cookie 8833927386536804216 [ 5054.155670] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5054.163759] LustreError: 179831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5054.468606] Lustre: server umount lustre-MDT0001 complete [ 5064.126779] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 5066.945518] Key type lgssc unregistered [ 5067.270778] LNet: 182869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5067.277205] LNetError: 182869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5067.293735] LNet: Removed LNI 192.168.206.155@tcp [ 5068.021296] Key type .llcrypt unregistered [ 5068.024384] Key type ._llcrypt unregistered [ 5090.302320] Key type ._llcrypt registered [ 5090.304704] Key type .llcrypt registered [ 5090.412798] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 5091.733602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5091.819037] alg: No test for adler32 (adler32-zlib) [ 5092.896756] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5093.189741] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 5094.856306] Key type lgssc registered [ 5096.115846] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5103.735881] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5103.761373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5105.198788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5109.680989] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5115.361873] LustreError: 184282:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5115.386049] LustreError: 184282:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5117.657982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5117.959547] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5122.248663] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5128.333356] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5132.893363] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5138.693964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5139.033705] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5144.807604] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5150.328899] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5151.230081] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5171.169284] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5171.175573] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5171.178173] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5171.203236] Lustre: Skipped 1 previous similar message [ 5176.295414] LustreError: 186721:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5176.309435] LustreError: 186721:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5176.405790] Lustre: server umount lustre-OST0000 complete [ 5181.413489] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5181.425296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5181.430520] Lustre: Skipped 2 previous similar messages [ 5185.891368] Lustre: server umount lustre-MDT0000 complete [ 5186.538822] LustreError: 184276:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5186.557470] LustreError: 184276:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5189.718269] LustreError: 184261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446400 with bad export cookie 6424816932543049851 [ 5189.723692] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5189.733458] LustreError: 184261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5190.111937] Lustre: server umount lustre-MDT0001 complete [ 5201.612612] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 5204.940190] Key type lgssc unregistered [ 5205.276919] LNet: 187842:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5205.300799] LNetError: 187842:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5205.351801] LNet: Removed LNI 192.168.206.155@tcp [ 5206.371826] Key type .llcrypt unregistered [ 5206.373937] Key type ._llcrypt unregistered [ 5229.439776] Key type ._llcrypt registered [ 5229.444387] Key type .llcrypt registered [ 5229.530825] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 5230.525284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5230.560367] alg: No test for adler32 (adler32-zlib) [ 5231.681764] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 5231.925826] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 5233.650989] Key type lgssc registered [ 5234.648574] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5241.251188] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5241.268526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5242.665197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5247.136964] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5252.069463] LustreError: 189258:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5252.080252] LustreError: 189258:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5254.702749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5255.165958] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5259.079610] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5265.369990] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5269.789636] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5275.124668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5275.444434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5280.941882] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5286.950717] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5287.739854] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5294.885646] Lustre: lustre-OST0000: Not available for connect from 192.168.206.55@tcp (stopping) [ 5296.106277] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5296.112768] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5296.129939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5298.155892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5298.158943] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5298.159871] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5299.927907] Lustre: server umount lustre-OST0000 complete [ 5307.189742] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8a0ad9558000.ost_server_uuid 50 [ 5308.867236] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5323.747529] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5323.755978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5323.765888] Lustre: Skipped 1 previous similar message [ 5323.774341] Lustre: Skipped 1 previous similar message [ 5328.091985] Lustre: server umount lustre-MDT0000 complete [ 5329.897485] LustreError: 189995:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5329.911587] LustreError: 189995:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5331.431400] LustreError: 189240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446542 with bad export cookie 15871566416809085983 [ 5331.442184] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5331.446951] LustreError: 189240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5331.790212] Lustre: server umount lustre-MDT0001 complete [ 5339.774500] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 10:42:30 (1788446550) [ 5346.004678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5346.474649] LustreError: 193186:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5346.575333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5350.927695] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5351.914083] LustreError: 193187:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5356.006789] LustreError: 193186:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5358.463270] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5358.740782] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5362.828753] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5370.286409] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5375.710461] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5382.008727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5382.341955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5388.653757] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5391.557781] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5395.706956] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5402.314421] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5402.520346] Lustre: lustre-OST0001: new disk, initializing [ 5402.524515] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5402.529714] Lustre: 196027:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5402.621671] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5407.789631] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5411.928523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5411.940082] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5412.032964] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5415.987882] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5419.493573] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5419.503707] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5419.520903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5419.525584] Lustre: Skipped 3 previous similar messages [ 5422.563800] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5424.310251] Lustre: server umount lustre-OST0000 complete [ 5426.943509] LustreError: 195005:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5429.394102] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5429.593795] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5433.615293] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5433.817688] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5443.053883] LustreError: 196561:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5443.087979] LustreError: 196561:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 5448.163839] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5448.170850] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5448.192446] Lustre: Skipped 2 previous similar messages [ 5448.192764] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5448.207921] Lustre: Skipped 4 previous similar messages [ 5453.584409] Lustre: server umount lustre-OST0001 complete [ 5458.402804] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5458.423977] Lustre: Skipped 1 previous similar message [ 5463.389392] Lustre: server umount lustre-MDT0000 complete [ 5466.862922] LustreError: 193167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446677 with bad export cookie 15871566416809087054 [ 5466.874038] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5466.875109] LustreError: 193167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5467.232615] Lustre: server umount lustre-MDT0001 complete [ 5489.643149] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 10:45:00 (1788446700) [ 5496.663446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5496.924227] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5496.978845] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5497.175969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5502.161504] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5511.370978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5511.482720] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5511.521887] Lustre: 199462:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5516.538540] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5524.838578] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5529.994649] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5536.010627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5536.222964] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5542.216063] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5545.647454] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5549.002238] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5555.211974] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5555.334228] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5560.744636] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5564.070640] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5567.140890] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5569.508189] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5569.521164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5569.526536] Lustre: Skipped 1 previous similar message [ 5569.534142] Lustre: Skipped 4 previous similar messages [ 5575.608334] Lustre: server umount lustre-OST0000 complete [ 5577.987901] LustreError: 202878:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5578.000072] LustreError: 202878:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 5594.594390] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5594.609583] LustreError: Skipped 1 previous similar message [ 5597.271671] Lustre: server umount lustre-OST0001 complete [ 5603.810834] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5603.839022] Lustre: Skipped 2 previous similar messages [ 5603.849601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5603.859941] Lustre: Skipped 7 previous similar messages [ 5607.583305] Lustre: server umount lustre-MDT0000 complete [ 5610.946900] LustreError: 199453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446821 with bad export cookie 15871566416809088461 [ 5610.948898] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5610.966594] LustreError: 199453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5611.384343] Lustre: server umount lustre-MDT0001 complete [ 5634.704837] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 10:47:25 (1788446845) [ 5648.873237] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 5657.842890] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 5687.894357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5688.081789] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5688.113806] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5688.192750] Lustre: lustre-MDT0000: new disk, initializing [ 5688.265035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5688.269822] Lustre: Skipped 3 previous similar messages [ 5688.287333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5692.964199] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5702.622231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5702.931828] Lustre: 207558:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5703.015783] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5703.021470] Lustre: Skipped 1 previous similar message [ 5703.134680] Lustre: lustre-MDT0001: new disk, initializing [ 5703.260033] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5703.273801] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5707.054629] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5715.053374] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5720.471397] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5727.310306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5727.496145] Lustre: lustre-OST0000: new disk, initializing [ 5727.499501] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5727.505200] Lustre: 209524:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5734.513819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5734.534708] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5734.687409] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5734.732865] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5742.995826] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5747.240729] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5747.393664] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5751.394608] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5751.590370] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5755.361248] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5755.368989] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5755.373617] LustreError: Skipped 3 previous similar messages [ 5755.386605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5755.395131] Lustre: Skipped 2 previous similar messages [ 5755.409485] Lustre: Skipped 3 previous similar messages [ 5759.690885] Lustre: server umount lustre-OST0000 complete [ 5769.467358] Lustre: server umount lustre-MDT0000 complete [ 5770.731748] LustreError: 208759:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5770.744244] LustreError: 208759:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 5772.715862] LustreError: 208352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446983 with bad export cookie 15871566416809090246 [ 5772.719139] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5772.728957] LustreError: 208352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5773.021652] Lustre: server umount lustre-MDT0001 complete [ 5780.597570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5784.957444] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5793.840705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5798.512248] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5806.031522] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5811.102139] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5817.401704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5817.651870] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5817.658823] Lustre: Skipped 4 previous similar messages [ 5823.405760] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5830.911573] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5835.556398] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5835.764397] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5840.757787] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5840.981846] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5845.475479] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5845.484583] LustreError: Skipped 1 previous similar message [ 5857.248503] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5857.492565] Lustre: server umount lustre-OST0000 complete [ 5863.666954] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5867.773505] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5895.549500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5901.212159] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5908.009628] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5911.282962] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5911.294854] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5911.299115] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5911.323439] Lustre: Skipped 1 previous similar message [ 5915.162234] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 5 sec [ 5921.765818] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5921.777271] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5921.779812] LustreError: Skipped 2 previous similar messages [ 5921.781150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5921.781155] Lustre: Skipped 10 previous similar messages [ 5921.803405] Lustre: Skipped 5 previous similar messages [ 5925.476167] Lustre: server umount lustre-OST0000 complete [ 5943.264127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5943.409000] Lustre: server umount lustre-MDT0000 complete [ 5946.337789] LustreError: 211739:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5946.366760] LustreError: 211739:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 5946.820492] LustreError: 211719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447157 with bad export cookie 15871566416809091128 [ 5946.826167] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5946.828863] LustreError: 211719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5947.148492] Lustre: server umount lustre-MDT0001 complete [ 5954.366866] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 10:52:44 (1788447164) [ 5960.373057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5964.939287] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5972.438423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5976.211298] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5982.430649] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5988.400098] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5994.140204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5999.534500] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6006.478855] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6011.726932] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6011.884315] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6015.749284] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6015.951123] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6022.125827] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6022.314578] Lustre: lustre-OST0001: new disk, initializing [ 6022.318513] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6022.327345] Lustre: 221076:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6027.733641] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6028.914622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6028.934704] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6029.080379] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6034.718372] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6038.145800] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6038.336958] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6041.861121] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6042.016303] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6049.887909] Lustre: server umount lustre-OST0001 complete [ 6054.835494] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6054.974195] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6059.067620] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6059.281285] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6086.457857] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6086.661336] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6086.665376] Lustre: Skipped 5 previous similar messages [ 6091.762988] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6091.778456] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6091.778907] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6092.871704] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6099.572246] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6101.160757] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6117.347684] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6117.498639] Lustre: server umount lustre-OST0001 complete [ 6135.796455] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6135.812691] LustreError: Skipped 2 previous similar messages [ 6140.800690] Lustre: server umount lustre-OST0000 complete [ 6150.690461] Lustre: server umount lustre-MDT0000 complete [ 6153.870190] LustreError: 221075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447364 with bad export cookie 15871566416809092094 [ 6153.871628] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6153.883017] LustreError: 221075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6160.429910] Lustre: server umount lustre-MDT0001 complete [ 6183.022658] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 10:56:33 (1788447393) [ 6189.136892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6189.374952] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6189.398309] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6193.772489] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6201.563305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6201.685702] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6201.727709] Lustre: 226316:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6206.540870] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6214.361512] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6219.908052] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6225.612954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6225.754120] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6230.550637] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6235.815771] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6243.402038] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6243.464857] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6243.556813] Lustre: lustre-OST0001: Not available for connect from 192.168.206.55@tcp (not set up) [ 6243.568242] Lustre: Skipped 26 previous similar messages [ 6248.564970] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6254.337450] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6255.506332] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6260.258376] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6260.457894] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6264.064613] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6264.261338] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6269.436558] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8a0ac7106000.ost_server_uuid 50 [ 6271.051390] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8a0ac7106000.ost_server_uuid in IDLE state after 0 sec [ 6272.096591] Lustre: Permanently deactivating lustre-OST0001 [ 6272.117050] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6275.918848] Lustre: Permanently reactivating lustre-OST0001 [ 6283.753477] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6283.767022] Lustre: Skipped 10 previous similar messages [ 6286.384078] Lustre: server umount lustre-OST0001 complete [ 6289.387144] LustreError: 228706:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6289.404417] LustreError: 228706:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 35 previous similar messages [ 6304.224513] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6304.390351] Lustre: server umount lustre-OST0000 complete [ 6313.830032] Lustre: server umount lustre-MDT0000 complete [ 6316.903773] LustreError: 227058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447527 with bad export cookie 15871566416809093382 [ 6316.909409] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6316.914557] LustreError: 227058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6317.222799] Lustre: server umount lustre-MDT0001 complete [ 6340.371334] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 10:59:10 (1788447550) [ 6346.967364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6347.159382] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6347.191275] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6347.196305] Lustre: Skipped 5 previous similar messages [ 6351.586312] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6360.730895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6360.799733] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6360.866334] Lustre: 233005:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6365.672789] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6373.122795] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6378.636942] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6384.649112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6384.855721] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6384.873231] Lustre: Permanently deactivating lustre-OST0000 [ 6384.882607] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6390.570705] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6391.336064] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6397.334086] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6403.641484] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6403.699072] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6409.691436] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6411.944802] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6416.328513] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6417.805768] Lustre: Permanently reactivating lustre-OST0000 [ 6417.821687] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6417.829144] Lustre: Skipped 2 previous similar messages [ 6426.706659] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 6426.719067] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 35 seconds. I think it's dead, and I am evicting it. exp 00000000e73beb7b, cur 1788447637 deadline 1788447632 last 1788447602 [ 6426.762904] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6432.738567] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6432.752376] LustreError: Skipped 5 previous similar messages [ 6436.798876] Lustre: server umount lustre-OST0001 complete [ 6446.421806] Lustre: server umount lustre-OST0000 complete [ 6456.070793] Lustre: server umount lustre-MDT0000 complete [ 6458.753660] LustreError: 232996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447669 with bad export cookie 15871566416809095034 [ 6458.756915] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6458.764861] LustreError: 232996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6459.004323] Lustre: server umount lustre-MDT0001 complete [ 6469.665337] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 6472.823107] Key type lgssc unregistered [ 6473.044560] LNet: 237915:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6473.050114] LNetError: 237915:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6473.064867] LNet: Removed LNI 192.168.206.155@tcp [ 6473.922964] Key type .llcrypt unregistered [ 6473.924451] Key type ._llcrypt unregistered [ 6493.384178] Key type ._llcrypt registered [ 6493.385837] Key type .llcrypt registered [ 6493.487860] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 6494.171356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6494.190204] alg: No test for adler32 (adler32-zlib) [ 6495.222245] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6495.476815] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 6497.152320] Key type lgssc registered [ 6498.063799] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6506.354094] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6506.364913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6507.656764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6511.614435] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6517.217548] LustreError: 239362:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6517.235312] LustreError: 239362:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6519.127533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6519.254871] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6519.309933] Lustre: 239352:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6519.323992] Lustre: Permanently deactivating lustre-MDT0001 [ 6519.337871] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6519.648850] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6521.650947] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6521.655380] LustreError: 239393:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6521.671242] LustreError: 239393:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6521.692072] LustreError: 239393:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 14, retries 1, failed: rc = -108 [ 6523.650257] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6530.039620] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6534.816545] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6541.107563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6541.387023] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6544.427583] LustreError: 241203:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6546.728456] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6549.479621] LustreError: 241201:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6552.568283] LustreError: 241203:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6552.588666] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6552.604384] LustreError: 241203:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6554.294283] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6557.670100] LustreError: 241203:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6557.721182] LustreError: 241203:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6561.772087] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6562.110900] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6567.445675] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6567.486273] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6567.931825] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6575.303197] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6579.701174] Lustre: Permanently reactivating lustre-MDT0001 [ 6579.735211] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6579.741466] Lustre: Skipped 2 previous similar messages [ 6585.130819] Lustre: lustre-MDT0001: Client 5ae00296-c489-4c9b-b773-7172460a7740 (at 192.168.206.55@tcp) reconnecting [ 6591.436080] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6593.193878] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6599.932316] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6600.182515] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6605.716387] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6605.963427] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6608.906189] Lustre: Permanently deactivating lustre-MDT0001 [ 6608.927893] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6608.939125] Lustre: Skipped 2 previous similar messages [ 6614.636067] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6623.725275] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6623.747330] Lustre: Skipped 1 previous similar message [ 6623.750189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6626.147427] Lustre: server umount lustre-MDT0000 complete [ 6628.835304] LustreError: 240105:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6629.741744] LustreError: 239342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447840 with bad export cookie 14326409026701354287 [ 6629.746867] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6629.748669] LustreError: 239342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6630.307535] Lustre: server umount lustre-MDT0001 complete [ 6634.169862] Lustre: server umount lustre-OST0000 complete [ 6645.446074] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 11:04:15 (1788447855) [ 6655.387178] Lustre: server umount lustre-OST0001 complete [ 6661.576223] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 6671.950824] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 6707.278224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6707.492730] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6707.503865] Lustre: Skipped 2 previous similar messages [ 6707.545904] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6707.719288] Lustre: lustre-MDT0000: new disk, initializing [ 6707.881836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6707.911539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6712.201818] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6722.063357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6722.187172] Lustre: 247621:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6722.257608] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6722.260942] Lustre: Skipped 1 previous similar message [ 6722.391921] Lustre: lustre-MDT0001: new disk, initializing [ 6722.502152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6722.537379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6722.551115] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6727.717192] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6737.342106] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6742.843877] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6749.122331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6749.318200] Lustre: lustre-OST0000: new disk, initializing [ 6749.321352] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6749.325853] Lustre: 249586:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6749.403778] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6756.011189] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6759.467690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6759.476261] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6759.586972] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6765.684350] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6767.874093] LustreError: 247627:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6776.559747] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6776.730812] Lustre: lustre-OST0001: new disk, initializing [ 6776.735285] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6776.744081] Lustre: 250704:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6777.968523] LustreError: 247627:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6778.000456] LustreError: 247628:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6782.719529] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6784.097500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6784.110714] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6784.212198] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6788.058081] LustreError: 247628:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6790.755423] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6792.202453] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6794.722740] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6794.729629] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6794.737799] LustreError: Skipped 1 previous similar message [ 6794.738468] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6794.738472] Lustre: Skipped 3 previous similar messages [ 6794.760957] Lustre: Skipped 3 previous similar messages [ 6799.848989] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6799.851066] Lustre: Skipped 2 previous similar messages [ 6800.505413] Lustre: server umount lustre-OST0001 complete [ 6800.704368] LustreError: 249569:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6800.726122] LustreError: 249569:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 6820.354983] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6820.359162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6820.370525] Lustre: Skipped 1 previous similar message [ 6823.705409] Lustre: server umount lustre-OST0000 complete [ 6830.049275] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6830.060827] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6830.080476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6830.089809] Lustre: Skipped 1 previous similar message [ 6833.896688] Lustre: server umount lustre-MDT0000 complete [ 6835.691459] LustreError: 247632:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6835.714909] LustreError: 247632:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 6837.560778] LustreError: 247611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448048 with bad export cookie 14326409026701356884 [ 6837.571449] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6837.572474] LustreError: 247611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6844.236800] Lustre: server umount lustre-MDT0001 complete [ 6854.844742] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 6857.758660] Key type lgssc unregistered [ 6858.110871] LNet: 252738:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6858.120030] LNetError: 252738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6858.143700] LNet: Removed LNI 192.168.206.155@tcp [ 6859.127816] Key type .llcrypt unregistered [ 6859.135573] Key type ._llcrypt unregistered [ 6905.436962] Key type ._llcrypt registered [ 6905.441150] Key type .llcrypt registered [ 6905.575195] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 6906.550858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6906.657695] alg: No test for adler32 (adler32-zlib) [ 6907.730924] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 6907.997951] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 6909.752203] Key type lgssc registered [ 6911.001258] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6919.244529] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6919.281825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6920.559341] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6920.625682] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6920.840499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6925.898509] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6935.672126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6935.755657] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6935.781765] Lustre: 254947:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6936.120687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6940.580954] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6950.117548] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6957.969160] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6965.488272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6965.706102] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6965.893344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6970.996385] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 6972.094662] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6978.908748] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7032.301588] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7032.315485] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7032.346835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7032.808153] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7036.267513] Lustre: server umount lustre-OST0000 complete [ 7040.668377] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7053.925330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7054.256407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7061.211545] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7069.369764] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7072.620015] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7072.629047] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7072.630639] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7072.648060] Lustre: Skipped 1 previous similar message [ 7113.697165] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7113.704654] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7113.707381] LustreError: Skipped 1 previous similar message [ 7113.734385] Lustre: Skipped 1 previous similar message [ 7113.734948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7113.734952] Lustre: Skipped 1 previous similar message [ 7114.954761] Lustre: server umount lustre-OST0000 complete [ 7118.817120] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7118.819045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7118.841661] Lustre: Skipped 2 previous similar messages [ 7118.853369] Lustre: Skipped 2 previous similar messages [ 7123.938046] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7123.957803] Lustre: Skipped 1 previous similar message [ 7124.992882] Lustre: server umount lustre-MDT0000 complete [ 7128.829661] LustreError: 254940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448339 with bad export cookie 12723869152881622761 [ 7128.834433] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7128.839456] LustreError: 254940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7129.449753] Lustre: server umount lustre-MDT0001 complete [ 7141.552506] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 7144.997296] Key type lgssc unregistered [ 7145.335627] LNet: 260371:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7145.345719] LNetError: 260371:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7145.360184] LNet: Removed LNI 192.168.206.155@tcp [ 7146.303905] Key type .llcrypt unregistered [ 7146.308876] Key type ._llcrypt unregistered [ 7171.837195] Key type ._llcrypt registered [ 7171.839439] Key type .llcrypt registered [ 7171.925317] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 7172.914606] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7172.950468] alg: No test for adler32 (adler32-zlib) [ 7174.073576] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7174.367382] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 7176.120386] Key type lgssc registered [ 7177.295318] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7185.294342] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7185.314534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7186.938530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7191.805773] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7196.131298] LustreError: 261786:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7196.175226] LustreError: 261786:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7201.250860] LustreError: 261787:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7202.414798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7202.881187] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7208.026851] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7215.742390] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7221.176158] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7228.502710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7229.033464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7231.109235] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7235.927279] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7243.808172] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7272.417468] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7272.423910] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7272.436520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7273.458069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7273.472089] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7273.478737] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7274.856625] Lustre: server umount lustre-OST0000 complete [ 7278.562380] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7278.572108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7278.581652] Lustre: Skipped 1 previous similar message [ 7278.595438] Lustre: Skipped 2 previous similar messages [ 7283.684814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7283.694016] Lustre: Skipped 1 previous similar message [ 7284.706796] Lustre: server umount lustre-MDT0000 complete [ 7288.188303] LustreError: 261767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448499 with bad export cookie 3857866355770607155 [ 7288.192349] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7288.205486] LustreError: 261767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7288.496217] Lustre: server umount lustre-MDT0001 complete [ 7297.464335] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 7299.883629] Key type lgssc unregistered [ 7300.193935] LNet: 266095:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7300.203462] LNetError: 266095:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7301.234415] LNet: Removed LNI 192.168.206.155@tcp [ 7302.067268] Key type .llcrypt unregistered [ 7302.069467] Key type ._llcrypt unregistered [ 7317.112785] Key type ._llcrypt registered [ 7317.115522] Key type .llcrypt registered [ 7317.208629] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 7318.244758] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7318.276995] alg: No test for adler32 (adler32-zlib) [ 7319.302749] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7319.477248] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 7321.161225] Key type lgssc registered [ 7322.352288] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7328.784240] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7328.799506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7330.172937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7334.031687] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7340.515216] LustreError: 267312:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7340.540085] LustreError: 267312:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7341.775942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7342.171830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7346.878324] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7355.009096] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7361.279662] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7368.490225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7368.871421] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7375.042667] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7375.725361] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7383.385451] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7400.929556] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7400.939680] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7400.970456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7402.977143] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7402.978761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7402.992733] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7403.022814] Lustre: Skipped 1 previous similar message [ 7405.470227] Lustre: server umount lustre-OST0000 complete [ 7413.217406] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7413.220088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7413.237971] Lustre: Skipped 1 previous similar message [ 7413.253586] Lustre: Skipped 1 previous similar message [ 7415.827323] Lustre: server umount lustre-MDT0000 complete [ 7418.341350] LustreError: 267311:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7418.356136] LustreError: 267311:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7419.219991] LustreError: 267292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448630 with bad export cookie 6049303232557252598 [ 7419.221747] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7419.236920] LustreError: 267292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7419.523377] Lustre: server umount lustre-MDT0001 complete [ 7432.739559] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 7436.211711] Key type lgssc unregistered [ 7436.549280] LNet: 270993:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7436.553762] LNetError: 270993:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7437.607706] LNet: Removed LNI 192.168.206.155@tcp [ 7438.601912] Key type .llcrypt unregistered [ 7438.605329] Key type ._llcrypt unregistered [ 7465.846886] Key type ._llcrypt registered [ 7465.849470] Key type .llcrypt registered [ 7465.988716] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 7467.172424] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7467.193933] alg: No test for adler32 (adler32-zlib) [ 7468.473410] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7468.761153] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 7470.448404] Key type lgssc registered [ 7472.256507] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7481.265488] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7481.277750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7482.995040] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7488.504212] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7492.579373] LustreError: 272411:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7492.607366] LustreError: 272411:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7496.900398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7497.356657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7502.204473] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7509.182207] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7514.660701] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7520.242623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7520.583790] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7527.073414] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7527.751661] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7534.863933] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7558.625948] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7558.639831] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7558.649030] LustreError: Skipped 1 previous similar message [ 7558.651632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7558.725250] Lustre: Skipped 1 previous similar message [ 7563.749538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7563.757180] Lustre: Skipped 2 previous similar messages [ 7564.737846] Lustre: server umount lustre-OST0000 complete [ 7573.987828] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7573.994362] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7574.009149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7574.027362] Lustre: Skipped 1 previous similar message [ 7575.633452] Lustre: server umount lustre-MDT0000 complete [ 7579.117515] LustreError: 272411:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7579.148689] LustreError: 272411:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7579.689608] LustreError: 272393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448790 with bad export cookie 4140697246538902673 [ 7579.697518] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7579.711428] LustreError: 272393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7580.173375] Lustre: server umount lustre-MDT0001 complete [ 7591.178085] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 7593.977654] Key type lgssc unregistered [ 7594.312668] LNet: 276722:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7594.321868] LNetError: 276722:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7594.346298] LNet: Removed LNI 192.168.206.155@tcp [ 7595.177830] Key type .llcrypt unregistered [ 7595.181313] Key type ._llcrypt unregistered [ 7614.769391] Key type ._llcrypt registered [ 7614.777394] Key type .llcrypt registered [ 7614.965967] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 7615.946672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7615.969816] alg: No test for adler32 (adler32-zlib) [ 7617.054740] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7617.261386] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 7619.026119] Key type lgssc registered [ 7620.310545] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7628.163575] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7628.192238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7629.730532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7634.228071] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7639.009547] LustreError: 277976:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7639.038265] LustreError: 277976:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7642.833099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7643.117889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7647.288512] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7654.402894] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7659.621936] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7665.491607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7665.803104] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7671.851296] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7674.040247] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7678.977979] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7694.817105] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7694.820663] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7694.825687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7694.853229] Lustre: Skipped 1 previous similar message [ 7694.870178] Lustre: Skipped 1 previous similar message [ 7696.370201] Lustre: server umount lustre-OST0000 complete [ 7699.940098] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7699.953844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7704.546863] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7704.556263] LustreError: Skipped 1 previous similar message [ 7704.560957] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7704.578860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7710.182215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7710.193784] Lustre: Skipped 3 previous similar messages [ 7714.272790] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7714.278507] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7714.304880] Lustre: Skipped 1 previous similar message [ 7714.527669] Lustre: server umount lustre-MDT0000 complete [ 7717.837129] LustreError: 277943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448928 with bad export cookie 145515040129904323 [ 7717.843971] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7717.859094] LustreError: 277943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7718.159763] Lustre: server umount lustre-MDT0001 complete [ 7729.415212] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 7733.258304] Key type lgssc unregistered [ 7733.629188] LNet: 281684:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7733.633994] LNetError: 281684:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7733.669640] LNet: Removed LNI 192.168.206.155@tcp [ 7734.853627] Key type .llcrypt unregistered [ 7734.857653] Key type ._llcrypt unregistered [ 7748.342278] Key type ._llcrypt registered [ 7748.345726] Key type .llcrypt registered [ 7748.512639] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 7764.807813] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 7779.011563] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 7779.951987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7780.099963] alg: No test for adler32 (adler32-zlib) [ 7781.204404] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 7781.404418] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 7783.056234] Key type lgssc registered [ 7784.211692] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7815.051799] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7815.090725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7816.460670] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7816.502621] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7816.629436] Lustre: lustre-MDT0000: new disk, initializing [ 7816.758744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7816.799849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7821.663818] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7831.551385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7831.664505] Lustre: 285238:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7831.708815] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7831.712440] Lustre: Skipped 1 previous similar message [ 7831.776936] Lustre: lustre-MDT0001: new disk, initializing [ 7831.868067] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7831.905872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7831.921441] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7836.443569] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7846.497747] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7852.513900] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7859.535212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7859.830988] Lustre: lustre-OST0000: new disk, initializing [ 7859.835815] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7859.845431] Lustre: 287207:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7859.964416] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7866.765225] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7869.521413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7869.534603] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7869.750471] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7876.403665] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7881.260263] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7881.447797] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7886.295638] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7886.479528] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7890.405045] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7890.406057] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7890.413728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7890.442889] Lustre: Skipped 1 previous similar message [ 7894.744323] Lustre: server umount lustre-OST0000 complete [ 7898.599916] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7898.609990] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7898.629237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7898.633243] Lustre: Skipped 1 previous similar message [ 7900.641346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7900.642766] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7904.812765] Lustre: server umount lustre-MDT0000 complete [ 7908.365860] LustreError: 287205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449119 with bad export cookie 11740329760851629239 [ 7908.369598] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7908.377581] LustreError: 287205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7908.644840] Lustre: server umount lustre-MDT0001 complete [ 7915.830230] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 11:25:26 (1788449126) [ 7923.355885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7923.884645] LustreError: 289619:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7923.893942] LustreError: 289619:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7923.934266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7928.483712] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7929.316130] LustreError: 289620:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7933.414173] LustreError: 289619:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7937.099477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7937.464980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7942.222900] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7950.222502] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7955.218909] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7961.432403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7961.727963] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7966.946869] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7973.667956] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7983.585812] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7983.595373] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7983.612041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7983.617865] Lustre: Skipped 1 previous similar message [ 7989.141672] Lustre: server umount lustre-OST0000 complete [ 7993.827874] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7993.836499] LustreError: Skipped 1 previous similar message [ 7993.839475] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7993.850874] Lustre: Skipped 1 previous similar message [ 7993.873585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7993.881477] Lustre: Skipped 2 previous similar messages [ 7998.724815] Lustre: server umount lustre-MDT0000 complete [ 7999.969754] LustreError: 289619:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7999.991904] LustreError: 289619:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8002.175835] LustreError: 289601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449213 with bad export cookie 11740329760851630121 [ 8002.183335] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8002.195217] LustreError: 289601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8002.633456] Lustre: server umount lustre-MDT0001 complete [ 8013.806882] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 8016.426556] Key type lgssc unregistered [ 8016.720107] LNet: 293156:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8016.725780] LNetError: 293156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8016.737802] LNet: Removed LNI 192.168.206.155@tcp [ 8017.630800] Key type .llcrypt unregistered [ 8017.635603] Key type ._llcrypt unregistered [ 8039.604790] Key type ._llcrypt registered [ 8039.607325] Key type .llcrypt registered [ 8039.674565] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 8051.796041] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 8052.911774] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8052.933935] alg: No test for adler32 (adler32-zlib) [ 8054.090635] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8054.322730] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 8056.024267] Key type lgssc registered [ 8057.261544] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8101.583150] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8101.597445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8102.841412] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8102.875944] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8102.952109] Lustre: lustre-MDT0000: new disk, initializing [ 8103.046092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8103.060614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8106.658746] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8114.918404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8115.110612] Lustre: 296961:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8115.189311] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8115.192809] Lustre: Skipped 1 previous similar message [ 8115.295702] Lustre: lustre-MDT0001: new disk, initializing [ 8115.443804] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8115.492639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8115.511418] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8119.574431] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8127.524226] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8131.846808] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8137.076515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8137.348229] Lustre: lustre-OST2710: new disk, initializing [ 8137.353726] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8137.368174] Lustre: 298929:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8137.449747] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8142.240699] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8144.432265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8144.442957] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8144.534731] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8150.097825] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8156.155776] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8156.379415] Lustre: lustre-OST03e8: new disk, initializing [ 8156.383585] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8156.392894] Lustre: 299994:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8156.476827] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8161.988912] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8164.961420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8164.968946] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8165.029086] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8167.903830] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8175.081832] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8175.272122] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8179.400837] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8179.624670] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8184.338645] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8184.534464] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8188.603553] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8188.727434] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8195.375808] Lustre: server umount lustre-MDT0000 complete [ 8196.076496] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8196.097525] Lustre: Skipped 3 previous similar messages [ 8199.374395] LustreError: 296952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449410 with bad export cookie 12203758509516193777 [ 8199.395660] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8199.400192] LustreError: 296952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8199.659982] Lustre: server umount lustre-MDT0001 complete [ 8213.399341] Lustre: server umount lustre-OST2710 complete [ 8226.969898] Lustre: server umount lustre-OST03e8 complete [ 8243.803355] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 8252.151663] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 8290.036201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8290.329403] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8290.374211] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8290.481635] Lustre: lustre-MDT0000: new disk, initializing [ 8290.631750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8290.648154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8294.936529] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8304.064780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8304.180474] Lustre: 305240:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8304.255452] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8304.260920] Lustre: Skipped 1 previous similar message [ 8304.351262] Lustre: lustre-MDT0001: new disk, initializing [ 8304.440325] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8304.474845] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8304.496703] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8308.527380] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8317.609251] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8323.012431] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8328.999458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8329.357827] Lustre: lustre-OST0000: new disk, initializing [ 8329.365398] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8329.371451] Lustre: 307209:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8329.490419] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8334.742380] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8339.500844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8339.508197] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8339.605646] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8342.152933] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8345.903293] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8346.087734] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8351.469996] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8351.823710] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8354.785404] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8354.793947] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8354.822377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8355.305630] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8360.356773] Lustre: server umount lustre-OST0000 complete [ 8365.539678] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8365.550821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8365.557424] Lustre: Skipped 2 previous similar messages [ 8365.570496] Lustre: Skipped 3 previous similar messages [ 8370.667955] LustreError: 305254:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8370.682211] LustreError: 305254:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 8370.758833] Lustre: server umount lustre-MDT0000 complete [ 8374.242692] LustreError: 305233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449585 with bad export cookie 12203758509516195457 [ 8374.257460] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8374.266959] LustreError: 305233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8374.594467] Lustre: server umount lustre-MDT0001 complete [ 8382.024288] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 11:33:12 (1788449592) [ 8383.528498] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8385.505373] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 11:33:15 (1788449595) [ 8389.536982] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 8420.079743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8420.274899] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8420.289333] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8420.428543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8424.438771] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8432.131952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8432.234491] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8432.289496] Lustre: 311066:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8436.764511] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8457.584625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8457.814860] Lustre: Denying initial registration attempt from nid 192.168.206.155@tcp, specified as failover [ 8457.830884] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8457.841461] LustreError: 312520:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 8457.853482] LustreError: 312520:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8457.863792] LustreError: 312520:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8457.925643] Lustre: server umount lustre-OST0000 complete [ 8457.928477] LustreError: 312520:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8468.452701] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8468.453050] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8468.478068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8468.490858] Lustre: Skipped 1 previous similar message [ 8470.381441] Lustre: server umount lustre-MDT0000 complete [ 8473.573877] LustreError: 312132:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8473.591525] LustreError: 312132:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8474.293637] LustreError: 311817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449685 with bad export cookie 12203758509516196332 [ 8474.294520] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8474.300644] LustreError: 311817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8474.628490] Lustre: server umount lustre-MDT0001 complete [ 8484.600709] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 8486.997996] Key type lgssc unregistered [ 8487.412817] LNet: 313643:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8487.424692] LNetError: 313643:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8487.442168] LNet: Removed LNI 192.168.206.155@tcp [ 8488.409902] Key type .llcrypt unregistered [ 8488.412899] Key type ._llcrypt unregistered [ 8499.378760] Key type ._llcrypt registered [ 8499.385246] Key type .llcrypt registered [ 8499.475795] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 8500.191685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8500.250130] alg: No test for adler32 (adler32-zlib) [ 8501.264266] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8501.474940] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 8503.120148] Key type lgssc registered [ 8504.253644] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8549.189910] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 8558.068765] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8558.096192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8559.473955] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8559.503372] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8559.685810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8563.863640] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8571.123089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8571.202940] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8571.229548] Lustre: 316410:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8571.514113] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8576.532950] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8583.162185] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8587.858100] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8606.315660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8606.528788] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8606.718254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8612.146428] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8618.635778] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8622.050106] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8622.064672] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8622.095038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8622.565148] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8622.614887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8626.755619] Lustre: server umount lustre-OST0000 complete [ 8632.801521] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8632.815989] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8632.846183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8632.853642] Lustre: Skipped 1 previous similar message [ 8636.734192] Lustre: server umount lustre-MDT0000 complete [ 8640.364363] LustreError: 316400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449851 with bad export cookie 10398550846886851332 [ 8640.373722] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8640.377976] LustreError: 316400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8640.675466] Lustre: server umount lustre-MDT0001 complete [ 8650.901519] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 8653.824888] Key type lgssc unregistered [ 8654.116612] LNet: 319984:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8654.120453] LNetError: 319984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8654.133561] LNet: Removed LNI 192.168.206.155@tcp [ 8655.129501] Key type .llcrypt unregistered [ 8655.131442] Key type ._llcrypt unregistered [ 8680.292478] Key type ._llcrypt registered [ 8680.295140] Key type .llcrypt registered [ 8680.391682] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 8681.743882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8681.945241] alg: No test for adler32 (adler32-zlib) [ 8683.117702] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8683.515891] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 8685.256232] Key type lgssc registered [ 8686.780529] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8695.056982] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8695.072361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8696.603339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8701.897702] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8706.021830] LustreError: 321403:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8706.046926] LustreError: 321403:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8709.972188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8710.290839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8715.488923] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8723.721812] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8729.258771] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8738.413032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8739.008196] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8745.676877] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8754.179183] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8765.927177] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8765.934255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8765.953361] Lustre: Skipped 2 previous similar messages [ 8765.959081] Lustre: Skipped 1 previous similar message [ 8767.269044] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.55@tcp (stopping) [ 8767.280337] Lustre: Skipped 1 previous similar message [ 8769.644718] Lustre: server umount lustre-MDT0000 complete [ 8771.044754] LustreError: 321398:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8771.064304] LustreError: 321398:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8772.347666] LustreError: 323991:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8773.470243] LustreError: 321382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449984 with bad export cookie 914885582014606864 [ 8773.482431] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8773.487600] LustreError: 321382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8776.161442] LustreError: 321399:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8776.163811] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8776.221758] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8779.946364] Lustre: server umount lustre-MDT0001 complete [ 8786.697760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8796.506478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8798.688258] LustreError: 324987:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 8798.689360] LustreError: 320711:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000007fdbf303 x1875325636092160/t0(0) o250->MGC192.168.206.155@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8799.075191] LustreError: 323236:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8799.099373] LustreError: 323236:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8799.207827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8799.290087] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 8803.664699] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8808.422562] LustreError: 325000:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8808.440737] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8808.485154] LustreError: 325000:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8813.264368] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8813.544448] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8813.558146] Lustre: Skipped 3 previous similar messages [ 8813.712190] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8818.112149] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8819.202229] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8825.122659] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8826.675675] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8832.058918] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8833.524945] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8839.148070] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8839.158030] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8839.174624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8845.073864] Lustre: server umount lustre-OST0000 complete [ 8849.377390] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8849.384306] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8849.400104] Lustre: Skipped 1 previous similar message [ 8849.405542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8849.413097] Lustre: Skipped 3 previous similar messages [ 8854.659366] Lustre: server umount lustre-MDT0000 complete [ 8855.015474] LustreError: 325005:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8855.034654] LustreError: 325005:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8858.183466] LustreError: 321383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450069 with bad export cookie 914885582014614494 [ 8858.185133] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8858.192543] LustreError: 321383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8858.551220] Lustre: server umount lustre-MDT0001 complete [ 8868.786491] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 8871.665352] Key type lgssc unregistered [ 8872.002647] LNet: 327674:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8872.024307] LNetError: 327674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8872.053237] LNet: Removed LNI 192.168.206.155@tcp [ 8873.005711] Key type .llcrypt unregistered [ 8873.009271] Key type ._llcrypt unregistered [ 8898.845596] Key type ._llcrypt registered [ 8898.851078] Key type .llcrypt registered [ 8898.990813] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 8899.873916] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8899.888405] alg: No test for adler32 (adler32-zlib) [ 8901.234984] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 8901.480173] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 8903.216215] Key type lgssc registered [ 8904.691722] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8912.064606] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8912.086424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8913.567775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8918.105772] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8923.118170] LustreError: 329089:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8923.161701] LustreError: 329089:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8926.641163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8926.939422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8931.053561] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8938.376449] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8944.003969] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8947.682790] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8947.692910] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8947.715766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8948.707051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8948.707792] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8951.843275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8951.852884] Lustre: Skipped 2 previous similar messages [ 8953.682476] Lustre: server umount lustre-MDT0000 complete [ 8956.706615] LustreError: 329070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450167 with bad export cookie 16562892623233422335 [ 8956.724441] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8957.009797] Lustre: server umount lustre-MDT0001 complete [ 8969.864150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8970.092764] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8970.154973] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8970.326635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8974.510787] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8983.084779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8983.171411] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8983.225443] Lustre: 331679:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8983.670180] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8989.266207] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8998.843937] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9005.587543] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9015.540183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9015.863400] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9015.885317] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9015.901923] LustreError: 333565:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 9015.912850] LustreError: 333565:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9015.925737] LustreError: 333565:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9016.063425] Lustre: server umount lustre-OST0000 complete [ 9016.072565] LustreError: 333565:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9026.095616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9026.378263] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9026.535664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9032.972637] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9033.897936] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9040.913281] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9049.195608] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9049.352158] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9049.429920] Lustre: lustre-OST0001: new disk, initializing [ 9049.433544] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9049.443486] Lustre: 335106:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9049.540085] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9056.147441] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9056.423542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9056.438497] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9056.612433] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9063.411344] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9066.978120] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9066.990851] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9067.020692] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9067.024830] Lustre: Skipped 1 previous similar message [ 9067.061936] Lustre: Skipped 1 previous similar message [ 9072.100066] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9072.129838] Lustre: Skipped 1 previous similar message [ 9073.016398] Lustre: server umount lustre-OST0001 complete [ 9077.229746] LustreError: 334440:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9079.784023] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9079.798969] LustreError: Skipped 1 previous similar message [ 9079.809482] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9080.289944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9080.298643] Lustre: Skipped 1 previous similar message [ 9082.343240] LustreError: 334440:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9082.359025] LustreError: 334440:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9084.551959] Lustre: server umount lustre-OST0000 complete [ 9091.051879] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9091.066108] LustreError: Skipped 1 previous similar message [ 9091.076057] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9091.090731] Lustre: Skipped 1 previous similar message [ 9094.900101] Lustre: server umount lustre-MDT0000 complete [ 9097.709958] LustreError: 332470:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9097.729848] LustreError: 332470:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9098.896423] LustreError: 331670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450309 with bad export cookie 16562892623233422888 [ 9098.903222] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9098.908738] LustreError: 331670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 9099.558394] Lustre: server umount lustre-MDT0001 complete [ 9111.473581] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 9114.803678] Key type lgssc unregistered [ 9115.162916] LNet: 337068:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9115.175073] LNetError: 337068:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9116.198885] LNet: Removed LNI 192.168.206.155@tcp [ 9117.127097] Key type .llcrypt unregistered [ 9117.129049] Key type ._llcrypt unregistered [ 9183.553249] Key type ._llcrypt registered [ 9183.555483] Key type .llcrypt registered [ 9183.732749] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 9198.789189] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 9200.216045] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9200.242542] alg: No test for adler32 (adler32-zlib) [ 9201.447829] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 9201.723035] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 9203.392662] Key type lgssc registered [ 9204.587737] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9237.704548] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9237.736152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9238.991532] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9239.031294] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9239.103733] Lustre: lustre-MDT0000: new disk, initializing [ 9239.164963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9239.185436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9243.629400] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9253.816783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9253.903457] Lustre: 341993:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9253.954041] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9253.959276] Lustre: Skipped 1 previous similar message [ 9254.027304] Lustre: lustre-MDT0001: new disk, initializing [ 9254.139079] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9254.172135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9254.184849] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9259.526466] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9269.337531] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9275.369194] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9282.298814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9282.665268] Lustre: lustre-OST0000: new disk, initializing [ 9282.669493] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9282.677402] Lustre: 343963:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9282.764841] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9288.746522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9288.755557] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9288.919362] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9289.318790] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9298.291573] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9302.453484] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9302.728513] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9307.583656] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9307.890437] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9314.788620] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9314.790727] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9314.809613] Lustre: Skipped 1 previous similar message [ 9314.811628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9316.385961] Lustre: server umount lustre-OST0000 complete [ 9320.930760] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9320.941865] LustreError: Skipped 1 previous similar message [ 9320.947228] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9320.967399] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9320.972146] Lustre: Skipped 1 previous similar message [ 9325.030696] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9325.035390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9325.052464] Lustre: Skipped 1 previous similar message [ 9326.489523] Lustre: server umount lustre-MDT0000 complete [ 9330.380204] LustreError: 341988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450541 with bad export cookie 11473077081657689743 [ 9330.394673] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9330.673896] Lustre: server umount lustre-MDT0001 complete [ 9339.952594] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 11:49:10 (1788450550) [ 9347.397867] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 11:49:17 (1788450557) [ 9356.633304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9357.193262] LustreError: 346720:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9357.222548] LustreError: 346720:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9357.264662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9362.125057] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9362.410398] LustreError: 346721:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9367.521423] LustreError: 346720:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9370.453955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9370.843296] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9375.282425] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9382.182708] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9386.907478] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9393.136534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9393.462993] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9399.490386] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9406.859786] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9414.991299] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9416.530996] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9417.979665] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9419.743153] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9421.570991] Lustre: Failing over lustre-MDT0000 [ 9421.962175] Lustre: server umount lustre-MDT0000 complete [ 9422.312148] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9422.321442] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9422.338603] LustreError: 346721:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9428.449632] LustreError: 346721:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9428.481308] LustreError: 346721:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 9438.690572] LustreError: 346720:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9438.701302] LustreError: 346720:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 9439.584749] Lustre: 340220:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788450634/real 1788450634] req@000000007f4f402a x1875326179284736/t0(0) o400->MGC192.168.206.155@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788450650 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9439.612480] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9440.400937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9449.953130] LustreError: 340216:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000006eecec6a x1875326179290112/t0(0) o250->MGC192.168.206.155@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9450.390378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9450.462828] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9450.752561] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9454.853223] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9455.603282] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9455.633588] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 9455.700755] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9463.944275] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9465.629727] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9469.014497] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9470.812654] Lustre: Failing over lustre-MDT0000 [ 9470.948776] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9470.951668] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9470.978767] Lustre: Skipped 2 previous similar messages [ 9470.980558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9471.001753] Lustre: Skipped 2 previous similar messages [ 9471.109620] Lustre: server umount lustre-MDT0000 complete [ 9471.287274] LustreError: 346717:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9471.312650] LustreError: 346717:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [ 9489.357841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9489.447589] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9489.681359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9489.742507] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9492.741767] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9493.759872] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9495.037926] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9495.061514] Lustre: Skipped 2 previous similar messages [ 9495.089884] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 9495.184511] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9504.748752] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9506.326518] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9515.491110] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9515.499382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9515.506676] Lustre: Skipped 3 previous similar messages [ 9515.520876] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9519.192096] Lustre: server umount lustre-MDT0000 complete [ 9520.615329] LustreError: 346717:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9520.629911] LustreError: 346717:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [ 9522.742314] LustreError: 348563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450733 with bad export cookie 11473077081657692459 [ 9522.758177] LustreError: 348563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9522.762785] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9523.043555] Lustre: server umount lustre-MDT0001 complete [ 9537.511981] Lustre: server umount lustre-OST0000 complete [ 9552.786455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9556.929686] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9565.375478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9565.847264] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9565.853899] Lustre: Skipped 1 previous similar message [ 9570.577099] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9578.232627] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9584.135568] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9591.453775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9598.026193] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9598.311678] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9605.821486] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9615.851177] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9615.852842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9615.871467] Lustre: Skipped 2 previous similar messages [ 9615.884780] Lustre: Skipped 2 previous similar messages [ 9622.006458] Lustre: server umount lustre-OST0000 complete [ 9632.056822] Lustre: server umount lustre-MDT0000 complete [ 9635.736030] LustreError: 353050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450846 with bad export cookie 11473077081657693208 [ 9635.740496] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9635.748293] LustreError: 353050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9636.442715] Lustre: server umount lustre-MDT0001 complete [ 9648.372100] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 9651.296594] Key type lgssc unregistered [ 9651.552870] LNet: 356602:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9651.561755] LNetError: 356602:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9651.577672] LNet: Removed LNI 192.168.206.155@tcp [ 9652.460760] Key type .llcrypt unregistered [ 9652.463647] Key type ._llcrypt unregistered [ 9677.326478] Key type ._llcrypt registered [ 9677.330702] Key type .llcrypt registered [ 9677.414613] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 9678.467755] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9678.529062] alg: No test for adler32 (adler32-zlib) [ 9679.523228] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [ 9679.766079] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 9681.464217] Key type lgssc registered [ 9682.703737] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9690.562767] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9690.582726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9692.055356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9695.040799] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9702.304731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9702.372515] LustreError: 358118:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9702.396904] LustreError: 358118:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9702.654565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9708.456669] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9716.059898] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9721.834258] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9728.625525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9728.910269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9734.596510] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9737.148783] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9741.654622] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9757.668051] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9757.671540] LustreError: 358115:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9757.694813] Lustre: Skipped 2 previous similar messages [ 9757.716315] LustreError: 358115:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9759.323907] Lustre: server umount lustre-MDT0000 complete [ 9762.786214] LustreError: 358119:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9762.820961] LustreError: 358119:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9763.431990] LustreError: 358098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450974 with bad export cookie 9771676007848599027 [ 9763.453351] LustreError: 358098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9763.454780] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9764.014493] Lustre: server umount lustre-MDT0001 complete [ 9778.778906] Lustre: server umount lustre-OST0000 complete [ 9791.289075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9791.818265] LustreError: 361850:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9791.873805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9797.091116] LustreError: 361851:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9797.232991] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9805.114442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9805.596884] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9810.118962] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9817.025425] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9822.404939] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9829.100310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9829.430112] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9834.430218] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9836.653290] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 9842.812919] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9856.993687] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9857.004345] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9857.027436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9861.907964] Lustre: server umount lustre-MDT0000 complete [ 9862.628287] LustreError: 364442:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9862.645352] LustreError: 364442:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9865.099367] LustreError: 361832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451076 with bad export cookie 9771676007848600140 [ 9865.103858] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9865.116904] LustreError: 361832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9865.374719] Lustre: server umount lustre-MDT0001 complete [ 9879.682241] Lustre: server umount lustre-OST0000 complete [ 9897.439489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9898.190058] LustreError: 365760:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9898.328152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9902.110861] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9910.063502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9915.202474] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9921.877289] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9926.847593] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9932.601352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9932.861433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9932.868738] Lustre: Skipped 1 previous similar message [ 9938.191386] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9943.107890] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 9945.275489] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9952.226604] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9952.234069] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9952.260063] Lustre: Skipped 2 previous similar messages [ 9952.268538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9952.279625] Lustre: Skipped 3 previous similar messages [ 9953.249743] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9956.815349] Lustre: server umount lustre-OST0000 complete [ 9961.953366] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9961.958492] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9961.963669] Lustre: Skipped 1 previous similar message [ 9961.967775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9961.983963] Lustre: Skipped 2 previous similar messages [ 9966.711097] Lustre: server umount lustre-MDT0000 complete [ 9968.609450] LustreError: 365761:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9968.626716] LustreError: 365761:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9970.155752] LustreError: 365741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451181 with bad export cookie 9771676007848601162 [ 9970.158226] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9970.172063] LustreError: 365741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9970.402839] Lustre: server umount lustre-MDT0001 complete [ 9979.997596] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 9983.117297] Key type lgssc unregistered [ 9983.446181] LNet: 369295:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9983.452934] LNetError: 369295:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9983.485340] LNet: Removed LNI 192.168.206.155@tcp [ 9984.245698] Key type .llcrypt unregistered [ 9984.248547] Key type ._llcrypt unregistered [10009.890671] Key type ._llcrypt registered [10009.893290] Key type .llcrypt registered [10010.038775] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [10011.115054] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10011.160230] alg: No test for adler32 (adler32-zlib) [10012.287457] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [10012.664728] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [10014.344133] Key type lgssc registered [10015.564470] Lustre: Echo OBD driver; http://www.lustre.org/ [10022.855042] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10022.866877] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10022.873718] LustreError: 370736:0:(osd_handler.c:8554:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10022.891530] LustreError: 370736:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10022.898888] LustreError: 370736:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10022.904950] LustreError: 370736:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10022.911464] LustreError: 370736:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [10029.479425] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10029.483944] LustreError: 371133:0:(osd_handler.c:8554:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10029.500743] LustreError: 371133:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10029.509988] LustreError: 371133:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10029.518804] LustreError: 371133:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10029.526887] LustreError: 371133:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [10046.878292] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [10049.780949] Key type lgssc unregistered [10050.137093] LNet: 372140:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10050.152485] LNetError: 372140:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10050.181265] LNet: Removed LNI 192.168.206.155@tcp [10051.187527] Key type .llcrypt unregistered [10051.191169] Key type ._llcrypt unregistered [10068.695774] Key type ._llcrypt registered [10068.698597] Key type .llcrypt registered [10068.827382] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [10083.268063] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [10084.205628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10084.287995] alg: No test for adler32 (adler32-zlib) [10085.491361] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [10085.719083] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [10087.416153] Key type lgssc registered [10088.352913] Lustre: Echo OBD driver; http://www.lustre.org/ [10121.856885] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10121.877675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10123.126797] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10123.150667] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10123.213446] Lustre: lustre-MDT0000: new disk, initializing [10123.279853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10123.298645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10127.742442] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10138.117787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10138.312258] Lustre: 375311:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10138.392023] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10138.403111] Lustre: Skipped 1 previous similar message [10138.548165] Lustre: lustre-MDT0001: new disk, initializing [10138.708511] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10138.744749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10138.753384] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10143.955349] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10153.948571] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10158.874651] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10164.375213] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10164.691582] Lustre: lustre-OST0000: new disk, initializing [10164.696312] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10164.705663] Lustre: 377275:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10164.813859] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10170.429075] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10174.017945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10174.035983] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10174.145104] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10179.975332] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10184.592243] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10184.811182] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10189.310595] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10189.611131] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10194.933056] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10194.947098] Lustre: Skipped 1 previous similar message [10194.950880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10194.954992] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10194.976970] Lustre: Skipped 1 previous similar message [10200.035644] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10200.042018] Lustre: Skipped 1 previous similar message [10205.162596] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10205.169778] Lustre: Skipped 1 previous similar message [10206.178604] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10206.345593] Lustre: server umount lustre-OST0000 complete [10210.287160] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10210.291496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10210.292072] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10210.292077] LustreError: Skipped 1 previous similar message [10210.318816] Lustre: Skipped 1 previous similar message [10215.395061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10215.402169] Lustre: Skipped 2 previous similar messages [10215.991364] Lustre: server umount lustre-MDT0000 complete [10219.260803] LustreError: 375303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451430 with bad export cookie 13874384660038060493 [10219.262365] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10219.269566] LustreError: 375303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10219.629212] Lustre: server umount lustre-MDT0001 complete [10228.468743] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10230.097438] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 12:04:00 (1788451440) [10236.237961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10236.679794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10241.065342] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10242.018807] LustreError: 379741:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10242.044306] LustreError: 379741:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10246.120303] LustreError: 379740:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10249.048247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10249.608869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10254.783982] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10263.404505] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10269.087715] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10275.183060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10275.586560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10281.635538] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10290.609664] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10297.525374] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10297.753783] Lustre: lustre-OST0001: new disk, initializing [10297.762165] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10297.773360] Lustre: 382578:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10297.863981] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10304.498463] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10307.205084] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10307.216988] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10307.323495] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10314.286949] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10321.195586] Lustre: lustre-OST0001: Not available for connect from 192.168.206.55@tcp (stopping) [10322.915160] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10322.944093] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10324.131861] Lustre: server umount lustre-OST0001 complete [10326.276475] LustreError: 381561:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10333.161831] LustreError: 381563:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10333.186488] LustreError: 381563:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10338.275667] LustreError: 381561:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10338.288338] LustreError: 381561:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10343.394615] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10343.403043] Lustre: Skipped 1 previous similar message [10343.406841] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10343.416108] Lustre: Skipped 3 previous similar messages [10347.302714] Lustre: server umount lustre-OST0000 complete [10352.098134] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10352.107616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10365.408138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10365.658406] Lustre: server umount lustre-MDT0000 complete [10368.884424] LustreError: 379722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451579 with bad export cookie 13874384660038061382 [10368.895953] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10368.898767] LustreError: 379722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10383.328121] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10383.645730] Lustre: server umount lustre-MDT0001 complete [10393.566897] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [10395.995659] Key type lgssc unregistered [10396.266585] LNet: 384571:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10396.276531] LNetError: 384571:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10396.299260] LNet: Removed LNI 192.168.206.155@tcp [10397.071411] Key type .llcrypt unregistered [10397.073703] Key type ._llcrypt unregistered [10441.359759] Key type ._llcrypt registered [10441.362511] Key type .llcrypt registered [10441.506269] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [10442.761991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10442.817134] alg: No test for adler32 (adler32-zlib) [10443.932096] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [10444.142329] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [10445.808201] Key type lgssc registered [10446.934440] Lustre: Echo OBD driver; http://www.lustre.org/ [10454.690830] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10454.725780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10456.105309] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10456.155145] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10456.341291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10461.108943] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10469.508837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10469.694882] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10469.748498] Lustre: 387351:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10469.871154] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10470.061597] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10474.203471] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10481.833331] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10487.267171] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10490.851583] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10490.853884] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10490.875914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10490.882339] Lustre: Skipped 1 previous similar message [10495.410875] Lustre: server umount lustre-MDT0000 complete [10498.825071] LustreError: 387343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451709 with bad export cookie 18041686857696580707 [10498.830918] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10498.833169] LustreError: 387343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10499.216473] Lustre: server umount lustre-MDT0001 complete [10507.615417] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 12:08:38 (1788451718) [10513.842423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10514.309260] LustreError: 389864:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10514.325960] LustreError: 389864:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10514.395517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10518.739639] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10519.532163] LustreError: 389865:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10524.647861] LustreError: 389864:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10526.791331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10527.259541] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10531.389417] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10539.021793] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10544.369526] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10550.302645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10550.444730] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10550.656020] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10556.051623] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10562.248167] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10570.927390] Lustre: Permanently deactivating lustre-OST0000 [10570.977181] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [10577.430286] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10579.592745] LustreError: 392850:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [10579.606419] LustreError: 392850:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10579.613672] LustreError: 392850:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10582.905439] LustreError: 392898:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10582.911863] LustreError: 392898:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10590.973944] Lustre: server umount lustre-OST0000 complete [10593.343526] LustreError: 393148:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10593.350864] LustreError: 393148:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10597.858566] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10597.871099] Lustre: Skipped 1 previous similar message [10597.877276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10597.882577] Lustre: Skipped 1 previous similar message [10602.978594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10602.989769] Lustre: Skipped 1 previous similar message [10605.129835] Lustre: server umount lustre-MDT0000 complete [10608.112919] LustreError: 390591:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10608.138022] LustreError: 390591:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10608.613367] LustreError: 389846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451819 with bad export cookie 18041686857696581288 [10608.623661] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10608.626919] LustreError: 389846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10619.114055] Lustre: server umount lustre-MDT0001 complete [10626.747357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10630.778188] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10632.849890] LustreError: 394307:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [10637.900565] Lustre: 394646:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.155@tcp [10639.824901] Lustre: server umount MGS complete [10650.400280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10650.844925] LustreError: 395370:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10650.867041] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10650.875586] Lustre: Skipped 1 previous similar message [10650.958656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10655.733668] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10661.353401] LustreError: 395370:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10661.394468] LustreError: 395370:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10664.926421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10665.193920] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10671.152946] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10681.263542] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10686.671403] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10688.741292] Lustre: Permanently reactivating lustre-OST0000 [10688.762120] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [10688.773328] Lustre: Skipped 2 previous similar messages [10718.691524] LustreError: 397212:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [10727.021844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10727.437180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10727.446078] Lustre: Skipped 1 previous similar message [10733.419557] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10739.846496] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10746.255764] Lustre: server umount lustre-OST0000 complete [10752.484244] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10752.490786] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10752.505579] Lustre: Skipped 1 previous similar message [10752.512375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10759.221474] Lustre: server umount lustre-MDT0000 complete [10762.484462] LustreError: 395352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451973 with bad export cookie 18041686857696582338 [10762.485257] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10762.494403] LustreError: 395352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10773.131967] Lustre: server umount lustre-MDT0001 complete [10782.596895] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [10785.359633] Key type lgssc unregistered [10785.630119] LNet: 399886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10785.637122] LNetError: 399886:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10785.655780] LNet: Removed LNI 192.168.206.155@tcp [10786.378457] Key type .llcrypt unregistered [10786.381182] Key type ._llcrypt unregistered [10801.258318] Key type ._llcrypt registered [10801.260817] Key type .llcrypt registered [10801.341031] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [10814.784610] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [10815.847548] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10815.865822] alg: No test for adler32 (adler32-zlib) [10816.851424] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [10817.030187] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [10818.688157] Key type lgssc registered [10819.834686] Lustre: Echo OBD driver; http://www.lustre.org/ [10849.926034] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10849.954428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10851.205449] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10851.252893] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10851.307277] Lustre: lustre-MDT0000: new disk, initializing [10851.356402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10851.372893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10855.654546] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10864.810973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10864.941383] Lustre: 403040:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10864.996863] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10865.000586] Lustre: Skipped 1 previous similar message [10865.094938] Lustre: lustre-MDT0001: new disk, initializing [10865.194736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10865.221817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10865.230869] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10868.982382] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10877.245734] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10882.201884] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10888.240266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10888.436566] Lustre: lustre-OST0000: new disk, initializing [10888.439466] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10888.444287] Lustre: 405008:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10888.502138] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10893.897500] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10898.495883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10898.500906] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10898.611049] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10903.208583] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10907.834864] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10908.040902] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10912.250545] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10912.444677] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10914.275130] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10914.308719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10914.311046] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10914.315827] Lustre: Skipped 1 previous similar message [10919.395489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10919.404857] Lustre: Skipped 1 previous similar message [10920.243201] Lustre: server umount lustre-OST0000 complete [10924.529729] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10924.542234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10924.544957] Lustre: Skipped 2 previous similar messages [10924.570626] Lustre: Skipped 1 previous similar message [10929.636346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10929.648513] Lustre: Skipped 1 previous similar message [10930.466604] Lustre: server umount lustre-MDT0000 complete [10933.655493] LustreError: 403034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452144 with bad export cookie 18394866383164099775 [10933.660526] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10933.661834] LustreError: 403034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10933.855500] Lustre: server umount lustre-MDT0001 complete [10939.947615] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [10941.146499] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 12:15:51 (1788452151) [10948.654500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10949.465442] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10954.184973] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10954.725333] LustreError: 407466:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10954.751141] LustreError: 407466:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10959.843770] LustreError: 407467:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10962.834473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10963.133291] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10967.260386] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10973.672091] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10978.494820] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10984.360872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10984.619569] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10990.156039] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10996.176375] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10997.607341] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11000.034085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11005.410225] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11005.415226] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11005.427736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11005.451872] Lustre: Skipped 1 previous similar message [11011.071629] Lustre: server umount lustre-OST0000 complete [11014.626204] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11014.633544] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11014.656345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11014.669507] Lustre: Skipped 3 previous similar messages [11020.282666] Lustre: server umount lustre-MDT0000 complete [11020.772900] LustreError: 407467:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11020.795318] LustreError: 407467:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11023.370821] LustreError: 407447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452234 with bad export cookie 18394866383164100657 [11023.372058] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11023.379497] LustreError: 407447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11023.611317] Lustre: server umount lustre-MDT0001 complete [11032.706683] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [11036.019743] Key type lgssc unregistered [11036.410857] LNet: 411195:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11036.431166] LNetError: 411195:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11036.446507] LNet: Removed LNI 192.168.206.155@tcp [11037.526094] Key type .llcrypt unregistered [11037.528969] Key type ._llcrypt unregistered [11059.802279] Key type ._llcrypt registered [11059.803612] Key type .llcrypt registered [11059.859685] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [11060.111338] Key type .llcrypt unregistered [11060.115704] Key type ._llcrypt unregistered [11076.040809] Key type ._llcrypt registered [11076.042821] Key type .llcrypt registered [11076.139172] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [11077.063962] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11077.102265] alg: No test for adler32 (adler32-zlib) [11078.186304] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11078.349500] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [11079.978106] Key type lgssc registered [11080.969909] Lustre: Echo OBD driver; http://www.lustre.org/ [11087.332151] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11087.341665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11088.665846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11092.720513] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11098.081848] LustreError: 413412:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11098.100046] LustreError: 413412:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11099.963774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11100.247476] LustreError: 413411:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11100.270455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11100.271955] LustreError: 413411:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11105.398212] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11105.772419] LustreError: 413412:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11105.794713] LustreError: 413412:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11110.887296] LustreError: 413411:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11110.908204] LustreError: 413411:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11111.641285] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11113.965678] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11117.500020] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11117.852444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11121.869867] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11133.422981] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11133.424208] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11133.430838] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11133.444946] Lustre: Skipped 1 previous similar message [11133.458449] Lustre: Skipped 1 previous similar message [11135.647224] Lustre: server umount lustre-OST0000 complete [11143.651445] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11143.654934] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11143.663324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11143.672960] Lustre: Skipped 1 previous similar message [11145.687790] Lustre: server umount lustre-MDT0000 complete [11148.512767] LustreError: 413393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452359 with bad export cookie 8498759816010097062 [11148.515800] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11148.527716] LustreError: 413393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11148.767637] Lustre: server umount lustre-MDT0001 complete [11158.177348] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [11160.709850] Key type lgssc unregistered [11160.985919] LNet: 416561:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11160.993828] LNetError: 416561:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11161.009602] LNet: Removed LNI 192.168.206.155@tcp [11162.017973] Key type .llcrypt unregistered [11162.020476] Key type ._llcrypt unregistered [11181.514280] Key type ._llcrypt registered [11181.515789] Key type .llcrypt registered [11181.600984] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [11182.313388] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11182.335883] alg: No test for adler32 (adler32-zlib) [11183.351677] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11183.531351] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [11185.170263] Key type lgssc registered [11185.976593] Lustre: Echo OBD driver; http://www.lustre.org/ [11191.566898] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11191.581109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11207.072879] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11212.192996] LustreError: 417929:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.155@tcp: failed processing log, type 4: rc = -110 [11238.816237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11242.897642] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11247.715331] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11252.533932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11252.826699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11252.867722] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11255.770354] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11259.370445] LustreError: 418965:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11259.382773] LustreError: 418965:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11261.954312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11262.381195] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11266.512995] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11272.872519] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11277.269831] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11283.426320] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11283.433050] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11283.447895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11285.388950] Lustre: server umount lustre-OST0000 complete [11288.036535] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11288.040915] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11288.043983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11288.057729] Lustre: Skipped 1 previous similar message [11293.153157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11293.160266] Lustre: Skipped 2 previous similar messages [11294.167136] Lustre: server umount lustre-MDT0000 complete [11296.711470] LustreError: 417936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452507 with bad export cookie 5997220504583605555 [11296.713941] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11296.720904] LustreError: 417936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11307.073765] Lustre: server umount lustre-MDT0001 complete [11318.126416] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [11320.968622] Key type lgssc unregistered [11321.334664] LNet: 421500:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11321.349123] LNetError: 421500:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11321.373309] LNet: Removed LNI 192.168.206.155@tcp [11322.515370] Key type .llcrypt unregistered [11322.520026] Key type ._llcrypt unregistered [11342.671483] Key type ._llcrypt registered [11342.673502] Key type .llcrypt registered [11342.756893] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [11343.534593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11343.625067] alg: No test for adler32 (adler32-zlib) [11344.684133] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11344.896878] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [11346.576329] Key type lgssc registered [11347.590387] Lustre: Echo OBD driver; http://www.lustre.org/ [11355.309581] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11355.332476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11356.756652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11361.181218] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11366.889404] LustreError: 422898:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11366.904963] LustreError: 422898:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11368.590094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11368.910532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11372.842958] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11379.267231] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11383.976690] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11388.917356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11389.223474] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11394.426765] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11399.869491] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11401.525589] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11409.889163] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11409.897069] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11409.908082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11410.706204] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.55@tcp (stopping) [11411.942420] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11411.942454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11411.957405] Lustre: Skipped 1 previous similar message [11411.962775] Lustre: Skipped 1 previous similar message [11413.437829] Lustre: server umount lustre-MDT0000 complete [11415.337740] LustreError: 425479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.55@tcp arrived at 1788452626 with bad export cookie 13659637700377994660 [11415.353829] LustreError: 425479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11416.890384] LustreError: 422879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452627 with bad export cookie 13659637700377993995 [11416.903123] LustreError: 422879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11416.903767] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11417.061152] LustreError: 422897:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11417.067990] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11417.082360] LustreError: 422897:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11417.116016] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11421.092894] LustreError: 422898:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11421.117403] LustreError: 422898:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11423.068919] Lustre: server umount lustre-OST0000 complete [11426.210033] LustreError: 422897:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11426.221820] LustreError: 422897:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11428.319173] Lustre: server umount lustre-MDT0001 complete [11437.158593] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [11439.280966] Key type lgssc unregistered [11439.539564] LNet: 426588:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11439.548855] LNetError: 426588:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11439.562725] LNet: Removed LNI 192.168.206.155@tcp [11440.198798] Key type .llcrypt unregistered [11440.201087] Key type ._llcrypt unregistered [11458.362079] Key type ._llcrypt registered [11458.366492] Key type .llcrypt registered [11458.455145] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [11459.363316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11459.410471] alg: No test for adler32 (adler32-zlib) [11460.493906] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11460.773316] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [11462.440269] Key type lgssc registered [11463.382114] Lustre: Echo OBD driver; http://www.lustre.org/ [11469.388301] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11469.401334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11470.725149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11474.059843] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11480.036645] LustreError: 427984:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11480.086434] LustreError: 427984:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11481.678404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11482.110918] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11486.811443] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11494.818051] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11501.148523] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11507.533902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11507.689548] LustreError: 429802:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11507.952816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11513.292845] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11515.389383] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:193) [11519.805513] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11525.612287] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11525.631760] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11525.642065] Lustre: Skipped 1 previous similar message [11526.934475] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.55@tcp (stopping) [11529.891454] Lustre: server umount lustre-MDT0001 complete [11530.727749] LustreError: 428711:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11530.747881] LustreError: 428711:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11535.848518] LustreError: 427979:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11535.865802] LustreError: 427979:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11540.963381] LustreError: 427984:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11540.985921] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11541.046624] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11541.057634] Lustre: Skipped 1 previous similar message [11541.081665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11543.282664] Lustre: server umount lustre-OST0000 complete [11548.148874] Lustre: server umount lustre-MDT0000 complete [11560.804172] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [11563.123564] Key type lgssc unregistered [11563.364755] LNet: 431669:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11563.371644] LNetError: 431669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11563.388266] LNet: Removed LNI 192.168.206.155@tcp [11564.210976] Key type .llcrypt unregistered [11564.212833] Key type ._llcrypt unregistered [11584.771735] Key type ._llcrypt registered [11584.776587] Key type .llcrypt registered [11584.867798] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [11598.088389] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [11599.789590] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11599.830986] alg: No test for adler32 (adler32-zlib) [11601.178369] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11601.791051] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [11603.584146] Key type lgssc registered [11605.071367] Lustre: Echo OBD driver; http://www.lustre.org/ [11643.418552] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [11657.557435] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11657.595203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11658.785945] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11658.814459] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11658.876854] Lustre: lustre-MDT0000: new disk, initializing [11658.946256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11658.970216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11662.407473] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11675.312250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11675.498652] Lustre: lustre-OST0000: new disk, initializing [11675.502561] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11675.505051] Lustre: Skipped 1 previous similar message [11675.512686] Lustre: 436683:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11675.573683] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11680.375068] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11684.925079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [11684.931887] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [11684.962518] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11687.284261] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11702.351575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11702.482527] Lustre: 435666:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11702.578833] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11702.660843] Lustre: lustre-MDT0001: new disk, initializing [11702.786173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11702.863327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [11702.877739] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [11707.885799] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11716.230124] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [11716.480823] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [11720.698127] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [11720.947529] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [11727.851056] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11727.881464] Lustre: Skipped 1 previous similar message [11727.881712] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11727.886466] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11727.900103] Lustre: Skipped 1 previous similar message [11729.673776] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.55@tcp (stopping) [11731.262803] Lustre: server umount lustre-MDT0001 complete [11738.081665] LustreError: 435677:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11738.111980] LustreError: 435677:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11743.206889] LustreError: 435673:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11743.228394] LustreError: 435673:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11748.322284] LustreError: 435676:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11748.322728] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11748.356219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11753.441353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11753.446128] LustreError: 435677:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11758.566742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11758.568208] LustreError: 435676:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11762.145140] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11762.308956] Lustre: server umount lustre-OST0000 complete [11772.718554] Lustre: server umount lustre-MDT0000 complete [11784.315410] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [11787.077846] Key type lgssc unregistered [11787.338966] LNet: 440434:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11787.348126] LNetError: 440434:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11787.368452] LNet: Removed LNI 192.168.206.155@tcp [11788.495996] Key type .llcrypt unregistered [11788.509149] Key type ._llcrypt unregistered [11825.623438] Key type ._llcrypt registered [11825.624816] Key type .llcrypt registered [11825.758903] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [11826.111233] Key type .llcrypt unregistered [11826.112915] Key type ._llcrypt unregistered [11846.471956] Key type ._llcrypt registered [11846.473781] Key type .llcrypt registered [11846.534549] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [11847.473420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11847.495620] alg: No test for adler32 (adler32-zlib) [11848.459778] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [11848.735863] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [11850.430685] Key type lgssc registered [11851.314440] Lustre: Echo OBD driver; http://www.lustre.org/ [11874.399635] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11874.458746] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11875.662899] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11875.691369] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11875.756787] Lustre: lustre-MDT0000: new disk, initializing [11875.813156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11875.831523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11880.005318] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11889.848576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11889.953864] Lustre: 443817:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11890.025466] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11890.031701] Lustre: Skipped 1 previous similar message [11890.118177] Lustre: lustre-MDT0001: new disk, initializing [11890.226470] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11890.263281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11890.276183] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11894.159258] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11903.017810] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11907.692586] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11914.085463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11914.383319] Lustre: lustre-OST0000: new disk, initializing [11914.387958] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11914.396200] Lustre: 445784:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11914.506674] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11919.905120] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11924.017308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11924.026194] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11924.128323] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11928.540287] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11936.227088] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11936.237709] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11936.260687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11939.809172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11939.811542] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11941.053206] Lustre: server umount lustre-MDT0000 complete [11944.351375] LustreError: 443809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453155 with bad export cookie 1851377329254152568 [11944.359627] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11944.366935] LustreError: 443809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11944.694997] Lustre: server umount lustre-MDT0001 complete [11957.133745] Lustre: server umount lustre-OST0000 complete [11965.709338] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 12:32:56 (1788453176) [11975.139843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11976.141165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11981.291990] LustreError: 447812:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11981.309125] LustreError: 447812:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11982.592022] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11986.406428] LustreError: 447811:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11991.524075] LustreError: 447812:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11991.762686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11992.147824] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11997.142569] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [12004.866850] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12011.168361] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12030.327191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12030.519330] Lustre: Found index 0 for lustre-OST0000, updating log [12030.777364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12037.103946] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [12045.541096] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12047.163886] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12057.577643] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12057.579657] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12057.604125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12057.630024] Lustre: Skipped 2 previous similar messages [12057.650353] Lustre: Skipped 3 previous similar messages [12062.493423] Lustre: server umount lustre-OST0000 complete [12078.265165] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12078.266566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12078.271816] Lustre: Skipped 1 previous similar message [12078.283546] Lustre: Skipped 3 previous similar messages [12083.680912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12083.689560] Lustre: Skipped 1 previous similar message [12084.741354] Lustre: server umount lustre-MDT0000 complete [12085.734675] LustreError: 447811:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12085.769788] LustreError: 447811:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12088.347484] LustreError: 447791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453299 with bad export cookie 1851377329254153961 [12088.354847] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12088.372620] LustreError: 447791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12089.044469] Lustre: server umount lustre-MDT0001 complete [12097.310600] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 12:35:07 (1788453307) [12098.592499] LNet: 1 local NIs in recovery (showing 1): 192.168.206.155@tcp [12716.721740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12717.198516] LustreError: 454226:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12717.213348] LustreError: 454226:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12717.272826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12722.266907] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [12722.661222] LustreError: 454227:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12727.789653] LustreError: 454226:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12730.445995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12730.724985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12735.568804] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [12742.915924] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12748.559184] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12768.122841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12768.358101] Lustre: Found index 0 for lustre-OST0000, updating log [12768.628486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12775.297297] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [12776.545810] LNet: 1 local NIs in recovery (showing 1): 192.168.206.155@tcp [12783.222939] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12785.792928] Lustre: server umount lustre-OST0000 complete [12791.078046] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12791.080530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12791.101559] Lustre: Skipped 1 previous similar message [12791.119000] Lustre: Skipped 1 previous similar message [12792.163517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12792.169561] Lustre: Skipped 1 previous similar message [12794.209857] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12794.213868] Lustre: Skipped 2 previous similar messages [12796.661427] Lustre: server umount lustre-MDT0000 complete [12797.284944] LustreError: 454955:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12797.308119] LustreError: 454955:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12800.410856] LustreError: 454206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454011 with bad export cookie 1851377329254155046 [12800.412996] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12800.431648] LustreError: 454206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12800.880431] Lustre: server umount lustre-MDT0001 complete [12831.682695] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [12842.450782] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [12879.732310] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 12:48:10 (1788454090) [12898.229493] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [12901.131525] Key type lgssc unregistered [12901.520577] LNet: 460843:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12901.530966] LNetError: 460843:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13030.568200] LNet: Removed LNI 192.168.206.155@tcp [13031.634937] Key type .llcrypt unregistered [13031.639221] Key type ._llcrypt unregistered [13049.713169] Key type ._llcrypt registered [13049.719079] Key type .llcrypt registered [13049.826278] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [13050.968978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13051.014734] alg: No test for adler32 (adler32-zlib) [13052.176251] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [13052.440706] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [13054.168142] Key type lgssc registered [13055.348722] Lustre: Echo OBD driver; http://www.lustre.org/ [13060.118422] LNet: Added LNI 192.168.206.155@tcp100 [8/256/0/180] [13061.127771] LNet: Added LNI 192.168.206.155@tcp101 [8/256/0/180] [13062.113040] LNet: Added LNI 192.168.206.155@tcp102 [8/256/0/180] [13063.146486] LNet: Added LNI 192.168.206.155@tcp103 [8/256/0/180] [13063.952709] LNet: Added LNI 192.168.206.155@tcp104 [8/256/0/180] [13064.825303] LNet: Added LNI 192.168.206.155@tcp105 [8/256/0/180] [13065.792176] LNet: Added LNI 192.168.206.155@tcp106 [8/256/0/180] [13066.660346] LNet: Added LNI 192.168.206.155@tcp107 [8/256/0/180] [13067.553152] LNet: Added LNI 192.168.206.155@tcp108 [8/256/0/180] [13068.383232] LNet: Added LNI 192.168.206.155@tcp109 [8/256/0/180] [13069.267864] LNet: Added LNI 192.168.206.155@tcp110 [8/256/0/180] [13070.235206] LNet: Added LNI 192.168.206.155@tcp111 [8/256/0/180] [13071.036809] LNet: Added LNI 192.168.206.155@tcp112 [8/256/0/180] [13071.844635] LNet: Added LNI 192.168.206.155@tcp113 [8/256/0/180] [13072.663492] LNet: Added LNI 192.168.206.155@tcp114 [8/256/0/180] [13073.442529] LNet: Added LNI 192.168.206.155@tcp115 [8/256/0/180] [13074.210777] LNet: Added LNI 192.168.206.155@tcp116 [8/256/0/180] [13075.110503] LNet: Added LNI 192.168.206.155@tcp117 [8/256/0/180] [13075.994408] LNet: Added LNI 192.168.206.155@tcp118 [8/256/0/180] [13076.912031] LNet: Added LNI 192.168.206.155@tcp119 [8/256/0/180] [13077.729815] LNet: Added LNI 192.168.206.155@tcp120 [8/256/0/180] [13078.610805] LNet: Added LNI 192.168.206.155@tcp121 [8/256/0/180] [13079.579217] LNet: Added LNI 192.168.206.155@tcp122 [8/256/0/180] [13080.558539] LNet: Added LNI 192.168.206.155@tcp123 [8/256/0/180] [13081.618483] LNet: Added LNI 192.168.206.155@tcp124 [8/256/0/180] [13082.560630] LNet: Added LNI 192.168.206.155@tcp125 [8/256/0/180] [13083.402747] LNet: Added LNI 192.168.206.155@tcp126 [8/256/0/180] [13084.307230] LNet: Added LNI 192.168.206.155@tcp127 [8/256/0/180] [13085.223944] LNet: Added LNI 192.168.206.155@tcp128 [8/256/0/180] [13086.098827] LNet: Added LNI 192.168.206.155@tcp129 [8/256/0/180] [13086.846035] LNet: Added LNI 192.168.206.155@tcp130 [8/256/0/180] [13087.609580] LNet: Added LNI 192.168.206.155@tcp131 [8/256/0/180] [13088.474799] LNet: Added LNI 192.168.206.155@tcp132 [8/256/0/180] [13089.119991] LNet: Added LNI 192.168.206.155@tcp133 [8/256/0/180] [13089.807822] LNet: Added LNI 192.168.206.155@tcp134 [8/256/0/180] [13090.453555] LNet: Added LNI 192.168.206.155@tcp135 [8/256/0/180] [13095.964694] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13095.975754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13097.089362] Lustre: 463875:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.155@tcp: can't add NID 192.168.206.155@tcp130: rc = -75 [13097.164280] LNetError: 463898:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.155 [13097.179072] LNetError: Protocol error connecting to 192.168.206.155@tcp109 at host 192.168.206.155:988: is it running a compatible version of Lustre? [13097.210962] LNetError: Skipped 13 previous similar messages [13097.308418] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13097.338973] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13097.425406] Lustre: lustre-MDT0000: new disk, initializing [13097.505444] LNet: 7 local NIs in recovery (showing 7): 192.168.206.155@tcp122, 192.168.206.155@tcp123, 192.168.206.155@tcp124, 192.168.206.155@tcp126, 192.168.206.155@tcp128, 192.168.206.155@tcp129, 192.168.206.155@tcp127 [13097.507907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13097.537497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13098.213757] LNetError: 461334:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.155 [13098.230677] LNetError: 461334:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 13 previous similar messages [13098.240345] LNetError: Protocol error connecting to 192.168.206.155@tcp113 at host 192.168.206.155:988: is it running a compatible version of Lustre? [13100.581096] LNetError: 463898:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.155 [13100.610645] LNetError: 463898:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 7 previous similar messages [13100.618077] LNetError: Protocol error connecting to 192.168.206.155@tcp122 at host 192.168.206.155:988: is it running a compatible version of Lustre? [13100.624977] LNetError: Skipped 9 previous similar messages [13101.742773] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [13104.682935] LNetError: 463903:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.155 [13104.698945] LNetError: 463903:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 4 previous similar messages [13104.713662] LNetError: Protocol error connecting to 192.168.206.155@tcp122 at host 192.168.206.155:988: is it running a compatible version of Lustre? [13104.732975] LNetError: Skipped 2 previous similar messages [13112.680132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13112.865722] LNetError: 461335:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.155 [13112.887280] LNetError: Protocol error connecting to 192.168.206.155@tcp122 at host 192.168.206.155:988: is it running a compatible version of Lustre? [13112.902058] Lustre: 463892:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13113.039698] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13113.056804] Lustre: Skipped 1 previous similar message [13113.209133] Lustre: lustre-MDT0001: new disk, initializing [13113.255269] Lustre: 464692:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.155@tcp: can't add NID 192.168.206.155@tcp130: rc = -75 [13113.266365] Lustre: 464692:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13113.450564] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13113.522419] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13113.549262] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13118.732975] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [13120.520442] Lustre: 465055:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.155@tcp: can't add NID 192.168.206.155@tcp130: rc = -75 [13120.525813] Lustre: 465055:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13127.203624] LNetError: 463896:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.155 [13127.223753] LNetError: Protocol error connecting to 192.168.206.155@tcp113 at host 192.168.206.155:988: is it running a compatible version of Lustre? [13127.400584] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13132.931333] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13138.884737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13139.286886] Lustre: lustre-OST0000: new disk, initializing [13139.291117] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13139.306809] Lustre: 465864:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13139.416984] Lustre: 465869:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.155@tcp: can't add NID 192.168.206.155@tcp130: rc = -75 [13139.425225] Lustre: 465869:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [13139.469046] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13144.611660] LNetError: 461333:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.155 [13144.625981] LNetError: 461333:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 13 previous similar messages [13144.636705] LNetError: Protocol error connecting to 192.168.206.155@tcp127 at host 192.168.206.155:988: is it running a compatible version of Lustre? [13144.641683] LNetError: Skipped 13 previous similar messages [13145.116928] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [13147.679689] Lustre: 466321:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.155@tcp: can't add NID 192.168.206.155@tcp130: rc = -75 [13147.687792] Lustre: 466321:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13147.768276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13147.784534] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13147.896902] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13156.302731] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13157.920119] LNet: 1 local NIs in recovery (showing 1): 192.168.206.155@tcp122 [13168.611863] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13168.613027] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13168.626755] LustreError: Skipped 1 previous similar message [13168.633100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13168.638543] Lustre: Skipped 1 previous similar message [13168.653075] Lustre: Skipped 1 previous similar message [13173.735461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13173.739313] Lustre: Skipped 1 previous similar message [13177.377911] LNetError: 463907:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.155 [13177.398870] LNetError: 463907:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 27 previous similar messages [13177.416668] LNetError: Protocol error connecting to 192.168.206.155@tcp118 at host 192.168.206.155:988: is it running a compatible version of Lustre? [13177.438424] LNetError: Skipped 27 previous similar messages [13178.853086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13178.863854] Lustre: Skipped 1 previous similar message [13179.360142] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13179.525860] Lustre: server umount lustre-OST0000 complete [13183.970339] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13183.974374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13183.990893] Lustre: Skipped 1 previous similar message [13184.001110] Lustre: Skipped 1 previous similar message [13189.091785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13189.097244] Lustre: Skipped 1 previous similar message [13189.486501] Lustre: server umount lustre-MDT0000 complete [13192.872388] LustreError: 467021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454403 with bad export cookie 11310979559678710034 [13192.875488] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13192.894189] LustreError: 467021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13193.477262] Lustre: server umount lustre-MDT0001 complete [13203.849139] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [13206.597343] Key type lgssc unregistered [13206.944178] LNet: 467784:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13206.955828] LNetError: 467784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13206.966358] LNet: Removed LNI 192.168.206.155@tcp [13208.041143] LNet: Removed LNI 192.168.206.155@tcp100 [13208.043317] LNetError: 467784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13208.053307] LNetError: 467784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13209.056170] LNet: Removed LNI 192.168.206.155@tcp101 [13209.061546] LNetError: 467784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13209.072690] LNet: Removed LNI 192.168.206.155@tcp102 [13210.144295] LNet: Removed LNI 192.168.206.155@tcp103 [13211.168138] LNet: Removed LNI 192.168.206.155@tcp104 [13211.173414] LNetError: 467784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13211.179822] LNetError: 467784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [13211.188936] LNet: Removed LNI 192.168.206.155@tcp105 [13212.256154] LNet: Removed LNI 192.168.206.155@tcp106 [13213.281047] LNet: Removed LNI 192.168.206.155@tcp107 [13214.304139] LNet: Removed LNI 192.168.206.155@tcp108 [13215.328322] LNet: Removed LNI 192.168.206.155@tcp109 [13215.344445] LNetError: 467784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13215.353100] LNetError: 467784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [13215.361496] LNet: Removed LNI 192.168.206.155@tcp110 [13216.416148] LNet: Removed LNI 192.168.206.155@tcp111 [13217.440226] LNet: Removed LNI 192.168.206.155@tcp112 [13218.464158] LNet: Removed LNI 192.168.206.155@tcp113 [13219.490053] LNet: Removed LNI 192.168.206.155@tcp114 [13220.512201] LNet: Removed LNI 192.168.206.155@tcp115 [13221.536175] LNet: Removed LNI 192.168.206.155@tcp116 [13222.560120] LNet: Removed LNI 192.168.206.155@tcp117 [13223.584144] LNet: Removed LNI 192.168.206.155@tcp118 [13223.591455] LNetError: 467784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13223.605184] LNetError: 467784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [13223.619541] LNet: Removed LNI 192.168.206.155@tcp119 [13224.672106] LNet: Removed LNI 192.168.206.155@tcp120 [13225.697417] LNet: Removed LNI 192.168.206.155@tcp121 [13225.699348] LNet: Removed LNI 192.168.206.155@tcp122 [13226.720551] LNet: Removed LNI 192.168.206.155@tcp123 [13227.744260] LNet: Removed LNI 192.168.206.155@tcp124 [13228.768162] LNet: Removed LNI 192.168.206.155@tcp125 [13229.792143] LNet: Removed LNI 192.168.206.155@tcp126 [13230.816145] LNet: Removed LNI 192.168.206.155@tcp127 [13231.840877] LNet: Removed LNI 192.168.206.155@tcp128 [13232.864400] LNet: Removed LNI 192.168.206.155@tcp129 [13232.874436] LNet: Removed LNI 192.168.206.155@tcp130 [13232.881372] LNet: Removed LNI 192.168.206.155@tcp131 [13232.883213] LNet: Removed LNI 192.168.206.155@tcp132 [13232.894688] LNet: Removed LNI 192.168.206.155@tcp133 [13232.905534] LNet: Removed LNI 192.168.206.155@tcp134 [13232.920229] LNet: Removed LNI 192.168.206.155@tcp135 [13234.348155] Key type .llcrypt unregistered [13234.350602] Key type ._llcrypt unregistered [13259.783621] Key type ._llcrypt registered [13259.786214] Key type .llcrypt registered [13259.877088] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [13273.402072] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [13274.655361] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13274.726772] alg: No test for adler32 (adler32-zlib) [13275.937927] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [13276.195159] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [13277.888233] Key type lgssc registered [13279.212596] Lustre: Echo OBD driver; http://www.lustre.org/ [13314.407972] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 12:55:24 (1788454524) [13331.292744] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [13333.874616] Key type lgssc unregistered [13334.286695] LNet: 471874:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13334.299824] LNetError: 471874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13334.318718] LNet: Removed LNI 192.168.206.155@tcp [13335.013354] Key type .llcrypt unregistered [13335.014827] Key type ._llcrypt unregistered [13350.861673] Key type ._llcrypt registered [13350.864173] Key type .llcrypt registered [13350.948068] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [13351.624514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13351.659500] alg: No test for adler32 (adler32-zlib) [13352.601980] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [13352.787355] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [13354.456172] Key type lgssc registered [13355.217757] Lustre: Echo OBD driver; http://www.lustre.org/ [13363.669800] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13363.703097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13364.895795] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13364.915317] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13364.993389] Lustre: lustre-MDT0000: new disk, initializing [13365.085633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13365.105478] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13368.921850] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [13379.368128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13379.477723] Lustre: 473088:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13379.524549] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13379.528869] Lustre: Skipped 1 previous similar message [13379.594277] Lustre: lustre-MDT0001: new disk, initializing [13379.684157] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13379.722496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13379.736674] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13383.659878] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [13394.485113] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13401.765690] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13410.018628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13410.328989] Lustre: lustre-OST0000: new disk, initializing [13410.331899] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13410.337405] Lustre: 475057:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13410.429903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13416.656524] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [13417.057395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13417.069693] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13417.181094] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13426.439633] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13430.202441] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing lnet_if_list [13436.047927] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [13436.082082] LNet: Added LNI 192.168.206.155@tcp10 [8/256/0/180] [13442.696077] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [13442.789975] LNet: Added LNI 192.168.206.155@tcp11 [8/256/0/180] [13447.610662] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [13447.664629] LNet: Added LNI 192.168.206.155@tcp12 [8/256/0/180] [13452.581960] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [13452.604708] LNet: Added LNI 192.168.206.155@tcp13 [8/256/0/180] [13457.645494] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [13457.667207] LNet: Added LNI 192.168.206.155@tcp14 [8/256/0/180] [13461.976949] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [13462.021824] LNet: Added LNI 192.168.206.155@tcp15 [8/256/0/180] [13467.036296] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [13467.075062] LNet: Added LNI 192.168.206.155@tcp16 [8/256/0/180] [13472.129424] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [13472.192577] LNet: Added LNI 192.168.206.155@tcp17 [8/256/0/180] [13477.389940] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [13477.421786] LNet: Added LNI 192.168.206.155@tcp18 [8/256/0/180] [13483.217966] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [13483.266510] LNet: Added LNI 192.168.206.155@tcp19 [8/256/0/180] [13488.282378] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [13488.337328] LNet: Added LNI 192.168.206.155@tcp20 [8/256/0/180] [13493.417955] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [13493.443887] LNet: Added LNI 192.168.206.155@tcp21 [8/256/0/180] [13499.544215] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [13499.599149] LNet: Added LNI 192.168.206.155@tcp22 [8/256/0/180] [13505.209614] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [13505.231953] LNet: Added LNI 192.168.206.155@tcp23 [8/256/0/180] [13510.811206] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [13510.864916] LNet: Added LNI 192.168.206.155@tcp24 [8/256/0/180] [13516.088175] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [13516.114638] LNet: Added LNI 192.168.206.155@tcp25 [8/256/0/180] [13521.203322] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [13521.235438] LNet: Added LNI 192.168.206.155@tcp26 [8/256/0/180] [13527.043700] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [13527.077167] LNet: Added LNI 192.168.206.155@tcp27 [8/256/0/180] [13535.953403] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [13536.004630] LNet: Added LNI 192.168.206.155@tcp28 [8/256/0/180] [13545.724538] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [13545.782102] LNet: Added LNI 192.168.206.155@tcp29 [8/256/0/180] [13555.420175] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [13555.488073] LNet: Added LNI 192.168.206.155@tcp30 [8/256/0/180] [13562.697896] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [13562.739185] LNet: Added LNI 192.168.206.155@tcp31 [8/256/0/180] [13570.445451] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [13570.466445] LNet: Added LNI 192.168.206.155@tcp32 [8/256/0/180] [13578.094556] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [13578.147443] LNet: Added LNI 192.168.206.155@tcp33 [8/256/0/180] [13586.144514] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [13586.165376] LNet: Added LNI 192.168.206.155@tcp34 [8/256/0/180] [13592.526687] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [13592.557242] LNet: Added LNI 192.168.206.155@tcp35 [8/256/0/180] [13599.446459] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [13599.467627] LNet: Added LNI 192.168.206.155@tcp36 [8/256/0/180] [13606.104782] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [13606.139554] LNet: Added LNI 192.168.206.155@tcp37 [8/256/0/180] [13613.714326] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [13613.738732] LNet: Added LNI 192.168.206.155@tcp38 [8/256/0/180] [13621.237319] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [13621.269140] LNet: Added LNI 192.168.206.155@tcp39 [8/256/0/180] [13629.269920] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [13629.301621] LNet: Added LNI 192.168.206.155@tcp40 [8/256/0/180] [13636.918265] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [13636.957897] LNet: Added LNI 192.168.206.155@tcp41 [8/256/0/180] [13644.163277] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [13644.193642] LNet: Added LNI 192.168.206.155@tcp42 [8/256/0/180] [13652.554956] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [13652.592610] LNet: Added LNI 192.168.206.155@tcp43 [8/256/0/180] [13659.151611] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [13659.187711] LNet: Added LNI 192.168.206.155@tcp44 [8/256/0/180] [13665.661243] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [13665.683373] LNet: Added LNI 192.168.206.155@tcp45 [8/256/0/180] [13674.328954] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [13674.389774] LNet: Added LNI 192.168.206.155@tcp46 [8/256/0/180] [13683.447256] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [13683.476467] LNet: Added LNI 192.168.206.155@tcp47 [8/256/0/180] [13692.255776] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [13692.309378] LNet: Added LNI 192.168.206.155@tcp48 [8/256/0/180] [13699.738917] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [13699.776701] LNet: Added LNI 192.168.206.155@tcp49 [8/256/0/180] [13708.782714] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [13708.811309] LNet: Added LNI 192.168.206.155@tcp50 [8/256/0/180] [13732.836130] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13732.836665] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13732.837831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13732.869624] Lustre: Skipped 1 previous similar message [13736.602903] Lustre: server umount lustre-MDT0000 complete [13740.098789] LustreError: 473081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454951 with bad export cookie 10974026668321641061 [13740.107181] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13740.429117] Lustre: server umount lustre-MDT0001 complete [13759.200299] Lustre: 472363:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788454954/real 1788454954] req@00000000fb2edba4 x1875330532035072/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788454970 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13759.221514] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13759.250206] Lustre: Skipped 1 previous similar message [13759.407532] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [13761.509031] Lustre: 472361:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788454956/real 1788454956] req@00000000aefee0d2 x1875330532035200/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788454972 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13764.576511] Lustre: 472360:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788454959/real 1788454959] req@00000000ae524b9d x1875330532035328/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788454975 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13766.636622] Lustre: 472362:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788454961/real 1788454961] req@00000000989dfe16 x1875330532035584/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788454977 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13769.940274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13771.872135] Lustre: 472362:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788454966/real 1788454966] req@00000000d40f135a x1875330532035712/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788454982 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13785.891754] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x984b931ca8f99cc2 [13785.908952] Lustre: MGC192.168.206.155@tcp: Connection restored to 0@lo (at 0@lo) [13786.235985] LustreError: 475049:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13786.258406] LustreError: 475049:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13786.341696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13788.386699] LustreError: 475050:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13788.391986] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13789.421381] LustreError: 475049:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13790.640940] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [13794.793625] LustreError: 475050:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13799.491599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13799.858112] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13804.183978] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [13804.991787] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13812.399119] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13819.146266] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13840.868316] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13840.873668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13840.886254] Lustre: Skipped 1 previous similar message [13840.889348] Lustre: Skipped 2 previous similar messages [13844.824780] Lustre: server umount lustre-OST0000 complete [13851.107803] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13851.114606] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13851.118581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13851.118589] Lustre: Skipped 1 previous similar message [13851.142123] Lustre: Skipped 1 previous similar message [13855.145276] Lustre: server umount lustre-MDT0000 complete [13856.225940] LustreError: 499318:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13856.248399] LustreError: 499318:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13858.638905] LustreError: 473081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455069 with bad export cookie 10974026668321643714 [13858.647523] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13858.656154] LustreError: 473081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [13859.032969] Lustre: server umount lustre-MDT0001 complete [13870.457363] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [13873.322186] Key type lgssc unregistered [13873.742614] LNet: 502112:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13873.750564] LNetError: 502112:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13873.756530] LNet: Removed LNI 192.168.206.155@tcp [13873.765868] LNet: Removed LNI 192.168.206.155@tcp10 [13873.768816] LNet: Removed LNI 192.168.206.155@tcp11 [13873.772234] LNet: Removed LNI 192.168.206.155@tcp12 [13873.774579] LNet: Removed LNI 192.168.206.155@tcp13 [13873.777317] LNet: Removed LNI 192.168.206.155@tcp14 [13873.779885] LNet: Removed LNI 192.168.206.155@tcp15 [13873.782711] LNet: Removed LNI 192.168.206.155@tcp16 [13873.785237] LNet: Removed LNI 192.168.206.155@tcp17 [13873.787572] LNet: Removed LNI 192.168.206.155@tcp18 [13873.790313] LNet: Removed LNI 192.168.206.155@tcp19 [13873.792924] LNet: Removed LNI 192.168.206.155@tcp20 [13873.795445] LNet: Removed LNI 192.168.206.155@tcp21 [13873.797307] LNet: Removed LNI 192.168.206.155@tcp22 [13873.799761] LNet: Removed LNI 192.168.206.155@tcp23 [13873.810188] LNet: Removed LNI 192.168.206.155@tcp24 [13873.812220] LNet: Removed LNI 192.168.206.155@tcp25 [13873.815604] LNet: Removed LNI 192.168.206.155@tcp26 [13873.818509] LNet: Removed LNI 192.168.206.155@tcp27 [13873.821572] LNet: Removed LNI 192.168.206.155@tcp28 [13873.824248] LNet: Removed LNI 192.168.206.155@tcp29 [13873.826356] LNet: Removed LNI 192.168.206.155@tcp30 [13873.829301] LNet: Removed LNI 192.168.206.155@tcp31 [13873.831675] LNet: Removed LNI 192.168.206.155@tcp32 [13873.834617] LNet: Removed LNI 192.168.206.155@tcp33 [13873.837247] LNet: Removed LNI 192.168.206.155@tcp34 [13873.839835] LNet: Removed LNI 192.168.206.155@tcp35 [13873.842529] LNet: Removed LNI 192.168.206.155@tcp36 [13873.846629] LNet: Removed LNI 192.168.206.155@tcp37 [13873.850566] LNet: Removed LNI 192.168.206.155@tcp38 [13873.854522] LNet: Removed LNI 192.168.206.155@tcp39 [13873.857352] LNet: Removed LNI 192.168.206.155@tcp40 [13873.861697] LNet: Removed LNI 192.168.206.155@tcp41 [13873.864548] LNet: Removed LNI 192.168.206.155@tcp42 [13873.870588] LNet: Removed LNI 192.168.206.155@tcp43 [13873.873906] LNet: Removed LNI 192.168.206.155@tcp44 [13873.876629] LNet: Removed LNI 192.168.206.155@tcp45 [13873.879718] LNet: Removed LNI 192.168.206.155@tcp46 [13873.882659] LNet: Removed LNI 192.168.206.155@tcp47 [13873.885320] LNet: Removed LNI 192.168.206.155@tcp48 [13873.888466] LNet: Removed LNI 192.168.206.155@tcp49 [13873.901849] LNet: Removed LNI 192.168.206.155@tcp50 [13874.777632] Key type .llcrypt unregistered [13874.781822] Key type ._llcrypt unregistered [13897.456031] Key type ._llcrypt registered [13897.457945] Key type .llcrypt registered [13897.677929] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [13915.925730] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [13917.261917] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13917.287430] alg: No test for adler32 (adler32-zlib) [13918.439758] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [13918.750110] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [13920.448170] Key type lgssc registered [13921.740454] Lustre: Echo OBD driver; http://www.lustre.org/ [13959.924017] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13959.967760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13961.302805] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13961.350828] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13961.478607] Lustre: lustre-MDT0000: new disk, initializing [13961.599402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13961.652457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13966.416481] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [13976.174061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13976.327658] Lustre: 505310:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13976.398229] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13976.404109] Lustre: Skipped 1 previous similar message [13976.487121] Lustre: lustre-MDT0001: new disk, initializing [13976.597942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13976.634776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13976.648140] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13980.977113] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [13991.615157] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13997.663658] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14006.805595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14007.042070] Lustre: *** cfs_fail_loc=90f, val=0*** [14007.130127] Lustre: lustre-OST0000: new disk, initializing [14007.133942] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14007.163498] Lustre: 507327:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14007.317410] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14014.366680] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [14017.049634] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14017.060073] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14017.177699] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14022.938524] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14032.355154] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14032.365729] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14032.401364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14032.891226] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14032.895980] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14032.951218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14037.085930] Lustre: server umount lustre-OST0000 complete [14043.105994] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14043.107425] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14043.134980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14043.166638] Lustre: Skipped 1 previous similar message [14048.293762] Lustre: server umount lustre-MDT0000 complete [14052.413628] LustreError: 507326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455263 with bad export cookie 17452070834772923478 [14052.419018] LustreError: 507326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14052.420330] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14052.786760] Lustre: server umount lustre-MDT0001 complete [14065.595876] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [14069.506569] Key type lgssc unregistered [14069.892518] LNet: 509234:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14069.899609] LNetError: 509234:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14069.921323] LNet: Removed LNI 192.168.206.155@tcp [14071.253564] Key type .llcrypt unregistered [14071.257130] Key type ._llcrypt unregistered [14093.484836] Key type ._llcrypt registered [14093.486590] Key type .llcrypt registered [14093.622698] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [14094.405860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14094.413066] alg: No test for adler32 (adler32-zlib) [14095.408912] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [14095.633032] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [14097.264190] Key type lgssc registered [14098.086824] Lustre: Echo OBD driver; http://www.lustre.org/ [14106.319633] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14106.365408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14108.002257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14111.625394] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [14118.379071] LustreError: 510504:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14118.406161] LustreError: 510504:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14121.206691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14121.839199] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14126.863584] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [14136.678201] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14142.641636] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14150.031663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14150.457956] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14157.381548] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [14165.213954] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14175.655761] loop0: detected capacity change from 0 to 204800000 [14175.911386] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14175.929028] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14175.961060] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14182.380220] loop0: detected capacity change from 0 to 204800000 [14182.675571] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14182.693122] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14182.708587] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14188.709936] loop0: detected capacity change from 0 to 204800000 [14195.058679] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14195.227902] Lustre: fs15246-OST0000: new disk, initializing [14195.233416] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [14195.241461] Lustre: 514099:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14195.344454] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14202.978338] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [14212.512928] loop1: detected capacity change from 0 to 204800000 [14217.656826] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14217.830764] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [14217.908526] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [14217.984131] Lustre: fs15246-MDT0000: new disk, initializing [14218.143067] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14218.161282] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14218.264717] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [14219.299241] Lustre: 515221:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000052cd1c0e x1875331310992128/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788455446 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [14223.869822] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [14228.390533] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14228.407440] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14228.429089] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14246.371241] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14246.406286] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14251.490241] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14252.210596] Lustre: server umount fs15246-MDT0000 complete [14256.615658] LustreError: 510499:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14261.433132] Lustre: server umount fs15246-OST0000 complete [14274.531955] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14274.547725] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14274.578798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14276.579833] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14276.590340] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14276.605566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14279.598333] Lustre: server umount lustre-OST0000 complete [14285.794790] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14285.813462] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14285.854530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14285.864191] Lustre: Skipped 2 previous similar messages [14289.971054] Lustre: server umount lustre-MDT0000 complete [14292.454049] LustreError: 510505:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14292.476519] LustreError: 510505:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14293.696349] LustreError: 510486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455504 with bad export cookie 4593839298792782813 [14293.701472] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14293.709208] LustreError: 510486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14294.083624] Lustre: server umount lustre-MDT0001 complete [14306.127884] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [14309.135557] Key type lgssc unregistered [14309.504789] LNet: 518323:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14309.520592] LNetError: 518323:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14309.536737] LNet: Removed LNI 192.168.206.155@tcp [14310.540155] Key type .llcrypt unregistered [14310.542651] Key type ._llcrypt unregistered [14331.157698] Key type ._llcrypt registered [14331.160179] Key type .llcrypt registered [14331.270379] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [14332.229264] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14332.275561] alg: No test for adler32 (adler32-zlib) [14333.412129] Lustre: Lustre: Build Version: 2.17.56_51_g2aa7cea [14333.644024] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [14335.328149] Key type lgssc registered [14336.930633] Lustre: Echo OBD driver; http://www.lustre.org/ [14378.885941] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [14388.608924] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [14422.499429] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14422.525183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14423.782187] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14423.805522] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14423.884754] Lustre: lustre-MDT0000: new disk, initializing [14423.971592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14423.988304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14428.389990] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [14438.478626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14438.581919] Lustre: 522789:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14438.638603] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14438.643541] Lustre: Skipped 1 previous similar message [14438.738885] Lustre: lustre-MDT0001: new disk, initializing [14438.837161] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14438.876356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14438.881808] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14442.878147] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [14451.200232] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14456.501265] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14463.211518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14463.462262] Lustre: lustre-OST0000: new disk, initializing [14463.466313] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14463.472053] Lustre: 524757:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14463.546367] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14469.506657] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [14472.341632] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14472.354412] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14472.418444] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14478.180926] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14482.902236] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [14483.211655] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [14489.034219] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [14489.318369] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [14493.155736] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14493.162065] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14493.165039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14493.174984] Lustre: Skipped 1 previous similar message [14493.190131] Lustre: Skipped 1 previous similar message [14498.010642] Lustre: server umount lustre-OST0000 complete [14503.394134] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14503.412061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14503.421298] Lustre: Skipped 1 previous similar message [14503.435316] Lustre: Skipped 1 previous similar message [14507.787416] Lustre: server umount lustre-MDT0000 complete [14511.395256] LustreError: 522781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455722 with bad export cookie 11318113708746200104 [14511.397373] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14511.413570] LustreError: 522781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14511.696246] Lustre: server umount lustre-MDT0001 complete [14538.829828] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [14549.624405] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [14580.522040] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14299 sec ========= 13:16:30 (1788455790) [14582.165720] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 13:16:32 (1788455792) === [14585.579503] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 13:16:35 (1788455795) === [14630.331827] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [14633.734640] Key type lgssc unregistered [14634.032819] LNet: 531404:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14634.040447] LNetError: 531404:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14634.052904] LNet: Removed LNI 192.168.206.155@tcp [14634.634616] Key type .llcrypt unregistered [14634.637256] Key type ._llcrypt unregistered