[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 566336062 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002200] x2apic enabled [ 0.003005] Switched APIC routing to physical x2apic. [ 0.004008] kvm-guest: setup PV IPIs [ 0.006000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007013] pid_max: default: 32768 minimum: 301 [ 0.009092] LSM: Security Framework initializing [ 0.010061] Yama: becoming mindful. [ 0.011041] SELinux: Initializing. [ 0.012071] *** VALIDATE selinux *** [ 0.020636] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024739] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025128] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026101] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027118] *** VALIDATE tmpfs *** [ 0.029052] *** VALIDATE proc *** [ 0.030019] *** VALIDATE cgroup *** [ 0.030796] *** VALIDATE cgroup2 *** [ 0.031215] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.032159] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.034027] Spectre V2 : User space: Vulnerable [ 0.035005] Speculative Store Bypass: Vulnerable [ 0.038729] debug: unmapping init [mem 0xffffffff8b059000-0xffffffff8b060fff] [ 0.040917] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.041612] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.042025] ... version: 2 [ 0.042819] ... bit width: 48 [ 0.043014] ... generic registers: 4 [ 0.043922] ... value mask: 0000ffffffffffff [ 0.044014] ... max period: 00007fffffffffff [ 0.045015] ... fixed-purpose events: 3 [ 0.045998] ... event mask: 000000070000000f [ 0.046300] rcu: Hierarchical SRCU implementation. [ 0.048320] smp: Bringing up secondary CPUs ... [ 0.049493] x86: Booting SMP configuration: [ 0.050033] .... node #0, CPUs: #1 #2 #3 [ 0.055313] smp: Brought up 1 node, 4 CPUs [ 0.056860] smpboot: Max logical packages: 1 [ 0.057016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.129483] node 0 deferred pages initialised in 69ms [ 0.133109] devtmpfs: initialized [ 0.134433] x86/mm: Memory block size: 128MB [ 0.138862] gcov: version magic: 0x41383552 [ 0.141314] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.142104] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.143313] pinctrl core: initialized pinctrl subsystem [ 0.144223] [ 0.144771] ************************************************************* [ 0.145017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.146025] ** ** [ 0.147017] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.148021] ** ** [ 0.149016] ** This means that this kernel is built to expose internal ** [ 0.150019] ** IOMMU data structures, which may compromise security on ** [ 0.151022] ** your system. ** [ 0.152016] ** ** [ 0.153018] ** If you see this message and you are not debugging the ** [ 0.154021] ** kernel, report this immediately to your vendor! ** [ 0.155016] ** ** [ 0.156016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.157015] ************************************************************* [ 0.158717] NET: Registered protocol family 16 [ 0.160209] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.163059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.165066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.168030] cpuidle: using governor menu [ 0.169472] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.170324] PCI: Using configuration type 1 for base access [ 0.172111] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.178132] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.180017] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.184167] cryptd: max_cpu_qlen set to 1000 [ 0.186206] ACPI: Added _OSI(Module Device) [ 0.188028] ACPI: Added _OSI(Processor Device) [ 0.189010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.191014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.196303] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.202549] ACPI: Interpreter enabled [ 0.204072] ACPI: PM: (supports S0 S3 S4 S5) [ 0.205038] ACPI: Using IOAPIC for interrupt routing [ 0.207180] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.210451] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.220417] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.222050] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.225024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.228142] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.234329] acpiphp: Slot [2] registered [ 0.236076] acpiphp: Slot [5] registered [ 0.236989] acpiphp: Slot [6] registered [ 0.238080] acpiphp: Slot [7] registered [ 0.239041] acpiphp: Slot [8] registered [ 0.240000] acpiphp: Slot [9] registered [ 0.240000] acpiphp: Slot [10] registered [ 0.241037] acpiphp: Slot [3] registered [ 0.241991] acpiphp: Slot [4] registered [ 0.242086] acpiphp: Slot [11] registered [ 0.243076] acpiphp: Slot [12] registered [ 0.244051] acpiphp: Slot [13] registered [ 0.244993] acpiphp: Slot [14] registered [ 0.246065] acpiphp: Slot [15] registered [ 0.246923] acpiphp: Slot [16] registered [ 0.248095] acpiphp: Slot [17] registered [ 0.249120] acpiphp: Slot [18] registered [ 0.250112] acpiphp: Slot [19] registered [ 0.252129] acpiphp: Slot [20] registered [ 0.253105] acpiphp: Slot [21] registered [ 0.254089] acpiphp: Slot [22] registered [ 0.254993] acpiphp: Slot [23] registered [ 0.256070] acpiphp: Slot [24] registered [ 0.256911] acpiphp: Slot [25] registered [ 0.258067] acpiphp: Slot [26] registered [ 0.259101] acpiphp: Slot [27] registered [ 0.261180] acpiphp: Slot [28] registered [ 0.263223] acpiphp: Slot [29] registered [ 0.265137] acpiphp: Slot [30] registered [ 0.267141] acpiphp: Slot [31] registered [ 0.270063] PCI host bridge to bus 0000:00 [ 0.271019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.273018] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.274020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.276018] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.278026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.279024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.281150] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.282817] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.285585] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.295017] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.302065] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.304022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.307024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.309020] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.312656] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.316198] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.319063] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.324432] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.332017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.344959] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.348908] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.355720] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.367023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.373017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.388024] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.397589] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.406020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.417023] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.436121] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.448221] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.460019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.470029] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.494021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.509407] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.528020] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.546024] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.574020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.593445] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.605018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.621024] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.671019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.691928] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.702021] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.713017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.735016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.748389] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.751615] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.755447] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.758410] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.761283] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.767033] iommu: Default domain type: Passthrough [ 0.769425] SCSI subsystem initialized [ 0.770084] ACPI: bus type USB registered [ 0.771068] usbcore: registered new interface driver usbfs [ 0.772074] usbcore: registered new interface driver hub [ 0.774084] usbcore: registered new device driver usb [ 0.775169] pps_core: LinuxPPS API ver. 1 registered [ 0.777057] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.780066] PTP clock support registered [ 0.782092] EDAC MC: Ver: 3.0.0 [ 0.785177] PCI: Using ACPI for IRQ routing [ 0.789913] NetLabel: Initializing [ 0.792109] NetLabel: domain hash size = 128 [ 0.794013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.796107] NetLabel: unlabeled traffic allowed by default [ 0.803089] vgaarb: loaded [ 0.805217] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.808015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.817661] clocksource: Switched to clocksource kvm-clock [ 0.942246] VFS: Disk quotas dquot_6.6.0 [ 0.944565] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.948293] *** VALIDATE ramfs *** [ 0.950211] *** VALIDATE hugetlbfs *** [ 0.952468] pnp: PnP ACPI init [ 0.955817] pnp: PnP ACPI: found 6 devices [ 0.973658] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.977194] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.979466] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.981595] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.983950] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.986522] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.989586] NET: Registered protocol family 2 [ 0.992209] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.997588] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.002095] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.007671] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.012786] TCP: Hash tables configured (established 65536 bind 65536) [ 1.016962] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.021263] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.024588] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.027787] NET: Registered protocol family 1 [ 1.031487] RPC: Registered named UNIX socket transport module. [ 1.033843] RPC: Registered udp transport module. [ 1.035800] RPC: Registered tcp transport module. [ 1.037719] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.040463] NET: Registered protocol family 44 [ 1.042903] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.046129] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.048726] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.051177] PCI: CLS 0 bytes, default 64 [ 1.052813] Unpacking initramfs... [ 2.475219] debug: unmapping init [mem 0xffff9b4e3cc54000-0xffff9b4e3ffbffff] [ 2.479258] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.481457] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.484635] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.013764] Initialise system trusted keyrings [ 3.015934] Key type blacklist registered [ 3.018504] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.026626] zbud: loaded [ 3.029603] *** VALIDATE nfs *** [ 3.030750] *** VALIDATE nfs4 *** [ 3.032377] pstore: using deflate compression [ 3.035674] Platform Keyring initialized [ 3.123263] NET: Registered protocol family 38 [ 3.126568] Key type asymmetric registered [ 3.128134] Asymmetric key parser 'x509' registered [ 3.130567] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.134443] io scheduler mq-deadline registered [ 3.136159] io scheduler kyber registered [ 3.138125] io scheduler bfq registered [ 3.140361] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.143796] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.147061] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.150279] ACPI: Power Button [PWRF] [ 3.155772] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.160595] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.173252] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.181724] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.195500] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.223272] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.252367] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.259234] Non-volatile memory driver v1.3 [ 3.261237] Linux agpgart interface v0.103 [ 3.299462] virtio_blk virtio1: [vda] 146016 512-byte logical blocks (74.8 MB/71.3 MiB) [ 3.303341] vda: detected capacity change from 0 to 74760192 [ 3.321980] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.325696] vdb: detected capacity change from 0 to 1073741824 [ 3.343066] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.347227] vdc: detected capacity change from 0 to 2621440000 [ 3.370592] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.373278] vdd: detected capacity change from 0 to 2621440000 [ 3.392921] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.397010] vde: detected capacity change from 0 to 4294967296 [ 3.411229] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.415403] vdf: detected capacity change from 0 to 4294967296 [ 3.426138] libphy: Fixed MDIO Bus: probed [ 3.431736] usbcore: registered new interface driver usbserial_generic [ 3.434606] usbserial: USB Serial support registered for generic [ 3.437166] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.441315] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.442715] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.444642] mousedev: PS/2 mouse device common for all mice [ 3.447451] rtc_cmos 00:05: RTC can wake from S4 [ 3.450892] rtc_cmos 00:05: registered as rtc0 [ 3.451033] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.453446] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.458335] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.459215] intel_pstate: CPU model not supported [ 3.463518] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.464215] hid: raw HID events driver (C) Jiri Kosina [ 3.468541] usbcore: registered new interface driver usbhid [ 3.470632] usbhid: USB HID core driver [ 3.472276] drop_monitor: Initializing network drop monitor service [ 3.474515] Initializing XFRM netlink socket [ 3.475955] NET: Registered protocol family 10 [ 3.478680] Segment Routing with IPv6 [ 3.480136] NET: Registered protocol family 17 [ 3.481972] mpls_gso: MPLS GSO support [ 3.488632] RAS: Correctable Errors collector initialized. [ 3.491709] AVX version of gcm_enc/dec engaged. [ 3.493792] AES CTR mode by8 optimization enabled [ 3.559549] sched_clock: Marking stable (3559530853, 0)->(4416787745, -857256892) [ 3.562087] registered taskstats version 1 [ 3.563979] Loading compiled-in X.509 certificates [ 3.565591] zswap: loaded using pool lzo/zbud [ 3.586098] Key type big_key registered [ 3.598495] Key type encrypted registered [ 3.599890] ima: No TPM chip found, activating TPM-bypass! [ 3.601792] ima: Allocated hash algorithm: sha1 [ 3.603439] ima: No architecture policies found [ 3.605136] evm: Initialising EVM extended attributes: [ 3.606987] evm: security.selinux [ 3.608214] evm: security.ima [ 3.609187] evm: security.capability [ 3.610385] evm: HMAC attrs: 0x1 [ 3.612702] rtc_cmos 00:05: setting system clock to 2026-08-31 04:55:41 UTC (1788152141) [ 3.618834] debug: unmapping init [mem 0xffffffff8c003000-0xffffffff8c1fffff] [ 3.621254] debug: unmapping init [mem 0xffffffff8ad82000-0xffffffff8b058fff] [ 3.625066] Write protecting the kernel read-only data: 28672k [ 3.627690] debug: unmapping init [mem 0xffffffff89403000-0xffffffff895fffff] [ 3.630596] debug: unmapping init [mem 0xffffffff89d14000-0xffffffff89dfffff] [ 3.664951] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.672741] systemd[1]: Detected virtualization kvm. [ 3.674441] systemd[1]: Detected architecture x86-64. [ 3.676251] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.696553] systemd[1]: No hostname configured. [ 3.698339] systemd[1]: Set hostname to . [ 3.700569] random: systemd: uninitialized urandom read (16 bytes read) [ 3.702950] systemd[1]: Initializing machine ID from random generator. [ 3.762746] random: ln: uninitialized urandom read (6 bytes read) [ 3.833080] random: systemd: uninitialized urandom read (16 bytes read) [ 3.836179] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.840783] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.845409] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.395073] device-mapper: uevent: version 1.0.3 [ 4.396922] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.028614] virtio_net virtio0 ens2: renamed from eth0 [ 5.059448] random: fast init done [ 5.160262] scsi host0: ata_piix [ 5.186152] scsi host1: ata_piix [ 5.189137] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.191773] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.592859] dracut-initqueue[580]: RTNETLINK answers: File exists [ 10.115212] random: crng init done [ 10.116524] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.324565] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.341962] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.620803] SELinux: Disabled at runtime. [ 11.681076] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.692779] systemd[1]: Detected virtualization kvm. [ 11.695270] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.228408] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.230665] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.238181] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.241707] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.246932] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.252546] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.256168] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... [ OK ] Stopped target Switch Root. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Remount Root and Kernel File Systems... [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting Huge Pages File System... [ OK [[ 12.411926] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS 0m] Created slice system-serial\x2dgetty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.661264] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.935227] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.945095] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.083908] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.093757] EDAC sbridge: Ver: 1.1.2 [ 14.552681] Key type dns_resolver registered [ 14.900969] NFS: Registering the id_resolver key type [ 14.902587] Key type id_resolver registered [ 14.903878] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg228-server login: [ 50.511677] libcfs: loading out-of-tree module taints kernel. [ 50.566332] Key type ._llcrypt registered [ 50.568881] Key type .llcrypt registered [ 50.691628] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 73.347826] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 75.397697] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 75.416913] alg: No test for adler32 (adler32-zlib) [ 75.843517] hrtimer: interrupt took 15376436 ns [ 77.105917] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 77.891674] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 79.615150] Key type lgssc registered [ 81.963750] Lustre: Echo OBD driver; http://www.lustre.org/ [ 100.245352] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 149.138346] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 165.135292] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 165.180832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 166.620957] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 166.661355] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 166.826103] Lustre: lustre-MDT0000: new disk, initializing [ 166.997916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 167.048540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 172.797295] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 189.798568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 189.938323] Lustre: 6509:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 189.985551] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 189.988708] Lustre: Skipped 1 previous similar message [ 190.081271] Lustre: lustre-MDT0001: new disk, initializing [ 190.141300] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 190.173181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 190.186071] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 195.324651] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 200.963514] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 211.494921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 211.969605] Lustre: lustre-OST0000: new disk, initializing [ 211.981848] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 211.998521] Lustre: 8448:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 212.110251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 213.369800] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 213.389021] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 213.548800] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 219.944986] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 236.779190] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 236.994412] Lustre: lustre-OST0001: new disk, initializing [ 237.003132] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 237.008232] Lustre: 9520:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 237.107708] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 245.282211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 245.290851] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 245.393196] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 245.473207] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 258.604670] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 266.135935] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 273.890327] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing check_logdir /tmp/testlogs/ [ 279.957797] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing yml_node [ 284.460301] Lustre: DEBUG MARKER: Client: 2.17.55.27 [ 287.099802] Lustre: DEBUG MARKER: MDS: 2.17.55.27 [ 290.006683] Lustre: DEBUG MARKER: OSS: 2.17.55.27 [ 292.445653] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 31 01:00:27 EDT 2026 [ 312.101208] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 313.784820] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 332.256501] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 332.258541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 332.271761] Lustre: Skipped 3 previous similar messages [ 332.280152] Lustre: Skipped 2 previous similar messages [ 337.074040] Lustre: server umount lustre-MDT0000 complete [ 341.497353] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788152479 with bad export cookie 11172394093285285354 [ 341.502242] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 341.511704] LustreError: 6500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 342.508975] LustreError: 6516:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 342.511116] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 342.533049] LustreError: 6516:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 342.542424] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 342.554450] Lustre: Skipped 1 previous similar message [ 342.565650] Lustre: Skipped 1 previous similar message [ 346.528305] LustreError: 6514:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 346.532962] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 346.569601] LustreError: 6514:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 346.579595] Lustre: Skipped 2 previous similar messages [ 348.203076] Lustre: server umount lustre-MDT0001 complete [ 353.349930] Lustre: server umount lustre-OST0000 complete [ 358.514354] Lustre: server umount lustre-OST0001 complete [ 366.559110] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 376.465461] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 418.897993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 419.121403] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 419.156204] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 419.231768] Lustre: lustre-MDT0000: new disk, initializing [ 419.327435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 419.338278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 424.522504] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 436.630475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 436.808056] Lustre: 15089:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 436.900705] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 436.912858] Lustre: Skipped 1 previous similar message [ 436.981425] Lustre: lustre-MDT0001: new disk, initializing [ 437.064571] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 437.098394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 437.109222] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 442.367948] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 452.963924] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 458.917473] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 466.310318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 466.603327] Lustre: lustre-OST0000: new disk, initializing [ 466.606892] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 466.612948] Lustre: 17059:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 466.763838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 468.589494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 468.596948] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 468.771161] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 472.473718] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 482.414452] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 487.009470] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 487.221917] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 492.445864] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 492.758772] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 499.690559] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 499.699423] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 499.723887] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 499.730109] LustreError: Skipped 1 previous similar message [ 499.752613] Lustre: Skipped 1 previous similar message [ 501.006762] Lustre: server umount lustre-OST0000 complete [ 504.818151] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 504.826784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 504.833434] Lustre: Skipped 2 previous similar messages [ 504.850065] Lustre: Skipped 1 previous similar message [ 515.043384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 515.050116] Lustre: Skipped 3 previous similar messages [ 519.135174] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 519.389243] Lustre: server umount lustre-MDT0000 complete [ 522.663554] LustreError: 15083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788152660 with bad export cookie 11172394093285286880 [ 522.672337] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 522.675445] LustreError: 15083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 523.144893] Lustre: server umount lustre-MDT0001 complete [ 526.990388] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 01:04:23 (1788152663) [ 529.346081] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 530.799759] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 01:04:27 (1788152667) [ 537.554861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 537.971415] LustreError: 19468:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 537.993985] LustreError: 19468:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 538.052806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 541.575087] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 543.200601] LustreError: 19469:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 543.227248] LustreError: 19469:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 547.318281] LustreError: 19468:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 551.647756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 557.625505] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 567.432638] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 573.694257] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 580.602614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 586.026848] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 591.746783] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 603.620515] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 603.638193] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 603.688141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 603.690525] Lustre: Skipped 2 previous similar messages [ 606.690333] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 609.349626] Lustre: server umount lustre-OST0000 complete [ 616.930135] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 616.950368] Lustre: Skipped 1 previous similar message [ 620.334134] Lustre: server umount lustre-MDT0000 complete [ 622.054406] LustreError: 19468:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 622.073719] LustreError: 19468:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 623.911670] LustreError: 21311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788152761 with bad export cookie 11172394093285287769 [ 623.917847] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 623.919943] LustreError: 21311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 624.370130] Lustre: server umount lustre-MDT0001 complete [ 634.881138] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 637.119572] Key type lgssc unregistered [ 637.384994] LNet: 23000:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 637.390947] LNetError: 23000:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 637.409283] LNet: Removed LNI 192.168.202.128@tcp [ 638.271164] Key type .llcrypt unregistered [ 638.276927] Key type ._llcrypt unregistered [ 663.501063] Key type ._llcrypt registered [ 663.511651] Key type .llcrypt registered [ 663.731313] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 666.275731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 666.297032] alg: No test for adler32 (adler32-zlib) [ 667.717370] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 668.563490] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 670.407168] Key type lgssc registered [ 672.627901] Lustre: Echo OBD driver; http://www.lustre.org/ [ 684.115287] loop: module loaded [ 684.193713] loop0: detected capacity change from 0 to 204800000 [ 684.451783] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 684.469463] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 684.861233] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 691.009514] loop0: detected capacity change from 0 to 204800000 [ 691.308601] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 691.320757] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 691.353918] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 698.377479] loop0: detected capacity change from 0 to 204800000 [ 703.866842] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 703.888815] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 705.134291] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 705.198227] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 705.272793] Lustre: lustre-MDT0000: new disk, initializing [ 705.337641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 705.353339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 709.868490] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 719.228145] loop1: detected capacity change from 0 to 204800000 [ 725.385919] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 725.722656] Lustre: lustre-OST07c6: new disk, initializing [ 725.726650] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 725.732603] Lustre: Skipped 1 previous similar message [ 725.738362] Lustre: 26191:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 725.812097] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 731.626874] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 734.364975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 734.385930] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 734.439973] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 736.571736] Lustre: 25180:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 736.577566] Lustre: 25180:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 736.582530] Lustre: 25180:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 736.591640] Lustre: 25180:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 736.596843] Lustre: 25180:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 736.602503] Lustre: 25180:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 750.095605] Lustre: 26198:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 750.110037] Lustre: 26198:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 750.118529] Lustre: 26198:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 750.133368] Lustre: 26198:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 750.141509] Lustre: 26198:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 750.151772] Lustre: 26198:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 755.388559] Lustre: 25179:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 755.395420] Lustre: 25179:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 755.406181] Lustre: 25179:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 755.412950] Lustre: 25179:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 755.418513] Lustre: 25179:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 755.429544] Lustre: 25179:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 755.434483] Lustre: 25179:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 755.447905] Lustre: 25179:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 755.451636] Lustre: 25179:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 755.456105] Lustre: 25179:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 755.460650] Lustre: 25179:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 755.466140] Lustre: 25179:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 758.722271] Lustre: 25178:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 758.736539] Lustre: 25178:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 758.743363] Lustre: 25178:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 758.748226] Lustre: 25178:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 758.752352] Lustre: 25178:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 758.757277] Lustre: 25178:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 758.761710] Lustre: 25178:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 758.766551] Lustre: 25178:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 758.770993] Lustre: 25178:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 758.775354] Lustre: 25178:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 758.780308] Lustre: 25178:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 758.784304] Lustre: 25178:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 763.384989] Lustre: 25178:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 763.393477] Lustre: 25178:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 763.404120] Lustre: 25178:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 763.409596] Lustre: 25178:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 763.414716] Lustre: 25178:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 763.420415] Lustre: 25178:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 763.424682] Lustre: 25178:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 763.429427] Lustre: 25178:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 763.435855] Lustre: 25178:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 763.441957] Lustre: 25178:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 763.452183] Lustre: 25178:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 763.458029] Lustre: 25178:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 774.713342] Lustre: 25180:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 774.730647] Lustre: 25180:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 774.744364] Lustre: 25180:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 774.757178] Lustre: 25180:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 774.762093] Lustre: 25180:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 774.767748] Lustre: 25180:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 774.777217] Lustre: 25180:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 774.785850] Lustre: 25180:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 774.793121] Lustre: 25180:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 774.798528] Lustre: 25180:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 774.805526] Lustre: 25180:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 774.813816] Lustre: 25180:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 826.343134] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 826.378335] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 827.982507] Lustre: server umount lustre-OST07c6 complete [ 838.155808] Lustre: server umount lustre-MDT0000 complete [ 858.490523] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 861.661926] Key type lgssc unregistered [ 861.940098] LNet: 28746:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 861.947824] LNetError: 28746:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 861.967351] LNet: Removed LNI 192.168.202.128@tcp [ 862.745641] Key type .llcrypt unregistered [ 862.752628] Key type ._llcrypt unregistered [ 880.303728] Key type ._llcrypt registered [ 880.306788] Key type .llcrypt registered [ 880.408939] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 881.626554] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 881.657276] alg: No test for adler32 (adler32-zlib) [ 882.818951] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 883.127488] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 884.815215] Key type lgssc registered [ 885.973504] Lustre: Echo OBD driver; http://www.lustre.org/ [ 893.838849] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 893.868587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 895.387454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 900.301631] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 905.707691] LustreError: 30015:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 905.726221] LustreError: 30015:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 910.008573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 910.496735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 915.674207] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 924.490681] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 931.221380] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 940.151563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 940.529874] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 940.754782] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 946.160131] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 948.340571] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 957.122521] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 975.840719] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 975.847665] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 975.851654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 975.856153] Lustre: Skipped 1 previous similar message [ 975.869491] Lustre: Skipped 1 previous similar message [ 980.971706] LustreError: 31840:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 981.308827] Lustre: server umount lustre-OST0000 complete [ 986.081512] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 986.087230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 986.097017] Lustre: Skipped 1 previous similar message [ 991.200614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 991.208609] Lustre: Skipped 2 previous similar messages [ 992.281558] Lustre: server umount lustre-MDT0000 complete [ 996.317334] LustreError: 29995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153134 with bad export cookie 13693624560608908740 [ 996.355944] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 996.900667] Lustre: server umount lustre-MDT0001 complete [ 1010.285121] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 1013.159375] Key type lgssc unregistered [ 1013.466533] LNet: 33754:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1013.475432] LNetError: 33754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1013.510234] LNet: Removed LNI 192.168.202.128@tcp [ 1014.441816] Key type .llcrypt unregistered [ 1014.444863] Key type ._llcrypt unregistered [ 1035.100864] Key type ._llcrypt registered [ 1035.103910] Key type .llcrypt registered [ 1035.288115] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 1037.226328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1037.264523] alg: No test for adler32 (adler32-zlib) [ 1038.333810] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1038.659229] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 1040.311174] Key type lgssc registered [ 1041.580404] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1049.769540] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1049.808251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1051.307210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1056.700394] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1061.867355] LustreError: 35025:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1061.882285] LustreError: 35025:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1065.516191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1065.940861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1070.885611] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1080.490460] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1087.023566] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1094.508766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1095.018799] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1097.100403] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1101.289493] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1109.348271] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1120.741348] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1120.745461] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1120.798997] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1120.814118] Lustre: Skipped 1 previous similar message [ 1125.872760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1125.880548] Lustre: Skipped 1 previous similar message [ 1126.313147] Lustre: server umount lustre-OST0000 complete [ 1130.977596] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1130.980912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1131.007135] Lustre: Skipped 2 previous similar messages [ 1136.099593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1136.110400] Lustre: Skipped 2 previous similar messages [ 1136.915548] Lustre: server umount lustre-MDT0000 complete [ 1141.218533] LustreError: 35754:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1141.242685] LustreError: 35754:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1141.858271] LustreError: 35006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153279 with bad export cookie 1430343322602322396 [ 1141.859931] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1141.872769] LustreError: 35006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1142.306857] Lustre: server umount lustre-MDT0001 complete [ 1156.598880] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 1160.615337] Key type lgssc unregistered [ 1161.035728] LNet: 38559:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1161.050932] LNetError: 38559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1162.086393] LNet: Removed LNI 192.168.202.128@tcp [ 1163.329417] Key type .llcrypt unregistered [ 1163.331276] Key type ._llcrypt unregistered [ 1193.468162] Key type ._llcrypt registered [ 1193.469537] Key type .llcrypt registered [ 1193.631955] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 1195.248739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1195.300903] alg: No test for adler32 (adler32-zlib) [ 1196.433690] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1196.742585] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 1198.455232] Key type lgssc registered [ 1200.088869] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1208.236547] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1208.256775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1209.827844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1215.579823] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1220.066516] LustreError: 39978:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1220.096905] LustreError: 39978:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1225.186289] LustreError: 39979:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1226.052379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1226.489571] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1230.329449] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1238.646701] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1245.951833] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1253.868463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1254.294896] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1255.350576] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1261.171970] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1269.806036] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1277.696395] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.28@tcp (stopping) [ 1277.923198] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1277.936786] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1282.028738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1282.033737] Lustre: Skipped 1 previous similar message [ 1282.036223] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1282.057287] Lustre: Skipped 1 previous similar message [ 1283.755985] Lustre: server umount lustre-MDT0000 complete [ 1287.149329] LustreError: 39979:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1287.178430] LustreError: 39979:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1287.760692] LustreError: 39957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153425 with bad export cookie 11224900678322552422 [ 1287.778249] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1287.917171] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.28@tcp (stopping) [ 1287.920105] Lustre: Skipped 3 previous similar messages [ 1292.256942] LustreError: 39974:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1292.258348] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1292.278898] LustreError: 39974:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1292.302137] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1294.276731] Lustre: server umount lustre-MDT0001 complete [ 1323.999145] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1324.197270] Lustre: server umount lustre-OST0000 complete [ 1342.776409] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 1345.793901] Key type lgssc unregistered [ 1346.155552] LNet: 43842:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1346.159497] LNetError: 43842:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1346.184383] LNet: Removed LNI 192.168.202.128@tcp [ 1347.437960] Key type .llcrypt unregistered [ 1347.439871] Key type ._llcrypt unregistered [ 1373.402394] Key type ._llcrypt registered [ 1373.405574] Key type .llcrypt registered [ 1373.527585] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 1375.145507] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1375.184322] alg: No test for adler32 (adler32-zlib) [ 1376.392995] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1376.645208] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 1378.351279] Key type lgssc registered [ 1379.560689] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1388.617394] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1388.646036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1390.190505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1394.897630] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1400.803949] LustreError: 45261:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1400.827406] LustreError: 45261:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1403.494665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1403.872262] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1409.524461] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1419.338950] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1426.984822] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1434.278137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1434.788425] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1435.844433] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1441.902449] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1449.965199] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1457.899235] Lustre: lustre-OST0000: Not available for connect from 192.168.202.28@tcp (stopping) [ 1459.680237] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1459.693167] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1459.699045] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1459.710585] Lustre: Skipped 1 previous similar message [ 1459.717952] Lustre: Skipped 1 previous similar message [ 1463.004555] Lustre: lustre-OST0000: Not available for connect from 192.168.202.28@tcp (stopping) [ 1463.428309] Lustre: server umount lustre-OST0000 complete [ 1484.259442] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1484.262618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1484.291948] Lustre: Skipped 1 previous similar message [ 1484.311628] Lustre: Skipped 1 previous similar message [ 1489.376180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1489.384064] Lustre: Skipped 1 previous similar message [ 1490.426230] Lustre: server umount lustre-MDT0000 complete [ 1493.817111] LustreError: 45242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153631 with bad export cookie 1878930090835614429 [ 1493.819365] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1493.828864] LustreError: 45242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1508.321387] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1508.535337] Lustre: server umount lustre-MDT0001 complete [ 1519.954345] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 1522.491575] Key type lgssc unregistered [ 1522.789637] LNet: 48973:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1522.796293] LNetError: 48973:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1522.810870] LNet: Removed LNI 192.168.202.128@tcp [ 1523.891509] Key type .llcrypt unregistered [ 1523.893066] Key type ._llcrypt unregistered [ 1546.415898] Key type ._llcrypt registered [ 1546.417473] Key type .llcrypt registered [ 1546.520418] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 1547.810605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1547.930670] alg: No test for adler32 (adler32-zlib) [ 1548.968394] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1549.240470] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 1550.961278] Key type lgssc registered [ 1552.139824] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1561.427972] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1561.462226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1563.069123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1568.270805] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1572.400797] LustreError: 50390:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1572.429034] LustreError: 50390:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1577.446143] LustreError: 50391:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1577.712810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1578.346705] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1584.541187] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1595.446484] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1602.948939] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1609.664280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1610.150717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1617.477906] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1625.062786] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1632.711944] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1642.077828] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1645.485352] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1658.002439] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-31 1h23m13s [ 1659.776530] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1662.734824] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.28@tcp (stopping) [ 1663.457291] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1663.467120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1663.481958] Lustre: Skipped 2 previous similar messages [ 1663.484876] Lustre: Skipped 1 previous similar message [ 1664.545331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1664.549239] Lustre: Skipped 4 previous similar messages [ 1667.801906] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.28@tcp (stopping) [ 1667.890850] Lustre: server umount lustre-MDT0000 complete [ 1670.625080] LustreError: 50390:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1670.652326] LustreError: 50390:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1671.086424] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1673.948960] LustreError: 51522:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1677.326075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1677.447399] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1677.728593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1677.814881] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1678.769610] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1682.655556] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1685.907832] Lustre: DEBUG MARKER: Wait for df (36720) ... [ 1687.864986] Lustre: DEBUG MARKER: done [ 1693.152682] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1693.163722] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1693.177506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1694.184690] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1699.227162] Lustre: server umount lustre-OST0000 complete [ 1704.418257] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1704.424677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1704.427373] Lustre: Skipped 1 previous similar message [ 1704.454204] Lustre: Skipped 3 previous similar messages [ 1709.582790] Lustre: server umount lustre-MDT0000 complete [ 1711.589470] LustreError: 50390:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1711.614600] LustreError: 50390:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1713.469698] LustreError: 50373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153851 with bad export cookie 11537283823739483209 [ 1713.471875] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1713.479493] LustreError: 50373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1713.809924] Lustre: server umount lustre-MDT0001 complete [ 1724.697280] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 1727.864971] Key type lgssc unregistered [ 1728.197871] LNet: 55336:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1728.205750] LNetError: 55336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1728.222703] LNet: Removed LNI 192.168.202.128@tcp [ 1729.473683] Key type .llcrypt unregistered [ 1729.479943] Key type ._llcrypt unregistered [ 1771.867488] Key type ._llcrypt registered [ 1771.870531] Key type .llcrypt registered [ 1771.988242] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 1773.823148] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1773.852923] alg: No test for adler32 (adler32-zlib) [ 1775.007946] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1775.320279] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 1777.111212] Key type lgssc registered [ 1778.419822] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1786.863614] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1786.885718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1788.089262] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1788.121106] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1788.227351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1793.597170] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1803.242601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1803.314898] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1803.347681] Lustre: 57546:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1803.667304] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1809.295782] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1817.814434] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1824.093910] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1831.629898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1831.934212] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1832.131940] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1839.119928] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1841.828301] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1848.650990] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1856.104595] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-31 1h26m31s [ 1857.826686] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1866.008224] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1868.568761] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1868.621420] LustreError: 59337:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1889.045433] Lustre: lustre-MDT0000: Client 69dd5f89-d04f-43f5-9ef1-1f93373a7b18 (at 192.168.202.28@tcp) reconnecting [ 1908.643909] LustreError: 59337:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1911.562393] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1914.675820] Lustre: DEBUG MARKER: done [ 1923.568033] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1923.590158] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1923.624731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1924.078089] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1929.047378] Lustre: server umount lustre-OST0000 complete [ 1934.309382] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1934.334104] Lustre: Skipped 1 previous similar message [ 1934.352056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1934.354891] Lustre: Skipped 2 previous similar messages [ 1935.407557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1935.423342] Lustre: Skipped 2 previous similar messages [ 1942.257898] Lustre: server umount lustre-MDT0000 complete [ 1947.557871] LustreError: 57537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154085 with bad export cookie 14418904824744169604 [ 1947.561123] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1947.564113] LustreError: 57537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1947.977785] Lustre: server umount lustre-MDT0001 complete [ 1960.821882] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 1964.064239] Key type lgssc unregistered [ 1964.459958] LNet: 61879:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1964.474180] LNetError: 61879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1964.497796] LNet: Removed LNI 192.168.202.128@tcp [ 1965.634091] Key type .llcrypt unregistered [ 1965.637709] Key type ._llcrypt unregistered [ 2019.218363] Key type ._llcrypt registered [ 2019.220712] Key type .llcrypt registered [ 2019.349828] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 2021.347924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2021.390845] alg: No test for adler32 (adler32-zlib) [ 2022.558228] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2022.926321] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 2024.559202] Key type lgssc registered [ 2025.604721] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2029.604827] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2029.639226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2031.059449] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2031.092255] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2031.428798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2032.571285] Lustre: Failing over lustre-MDT0000 [ 2032.850947] Lustre: server umount lustre-MDT0000 complete [ 2041.331030] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 01:29:36 (1788154176) [ 2049.196297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2049.579677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2054.294043] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2064.378357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2064.560895] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2064.643057] Lustre: 64683:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2065.051463] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2070.817222] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2081.486429] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2087.331573] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2094.602132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2094.854980] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2095.251419] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2102.655499] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2103.033913] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2111.733593] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2118.314332] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2126.306806] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2126.329326] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2126.358965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2128.868341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2128.871254] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2130.967325] Lustre: server umount lustre-MDT0000 complete [ 2135.483628] LustreError: 64674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154273 with bad export cookie 11622910440581204870 [ 2135.493057] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2135.495949] LustreError: 64674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2135.909446] Lustre: server umount lustre-MDT0001 complete [ 2139.958609] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2145.705918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2154.464033] Lustre: 63604:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788154276/real 1788154276] req@000000001c85c654 x1875015536520192/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788154292 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2154.507119] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2154.524623] Lustre: Skipped 1 previous similar message [ 2156.257594] Lustre: 63605:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788154278/real 1788154278] req@00000000dde378d9 x1875015536520320/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788154294 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2156.407208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2160.610039] Lustre: 63607:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788154282/real 1788154282] req@00000000c01ba80a x1875015536520448/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788154298 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2160.676801] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xa14cdf7703059469 [ 2160.697574] Lustre: MGC192.168.202.128@tcp: Connection restored to 0@lo (at 0@lo) [ 2161.066981] LustreError: 66531:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2161.088400] LustreError: 66531:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2161.213257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2161.306752] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2163.237794] LustreError: 66532:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2164.263163] LustreError: 66531:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2164.281351] LustreError: 66531:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2166.303509] Lustre: 63604:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788154288/real 1788154288] req@00000000ccaa8308 x1875015536520832/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788154304 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2166.323706] Lustre: 63604:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2166.802858] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2169.313553] LustreError: 68453:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2169.327957] LustreError: 68453:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2174.434155] LustreError: 68450:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2174.449614] LustreError: 68450:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2177.456160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2177.928131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2183.156948] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2183.180283] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2184.245976] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2192.397358] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2198.767720] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2208.738180] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2208.758185] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2208.771155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2208.782285] Lustre: Skipped 2 previous similar messages [ 2212.894321] Lustre: server umount lustre-MDT0000 complete [ 2214.881323] LustreError: 66532:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2214.901123] LustreError: 66532:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2217.369682] LustreError: 64674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154355 with bad export cookie 11622910440581207145 [ 2217.383076] LustreError: 64674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2217.391562] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2217.847776] Lustre: server umount lustre-MDT0001 complete [ 2226.126114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2235.360742] Lustre: 63607:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788154357/real 1788154357] req@0000000089ba4a7c x1875015536549760/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788154373 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2235.377328] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2235.397273] Lustre: Skipped 2 previous similar messages [ 2237.709160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2244.004693] LustreError: 66531:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2244.105291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2244.176452] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2245.023146] Lustre: 63605:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788154366/real 1788154366] req@00000000cc0266dc x1875015536550272/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788154382 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2245.067616] Lustre: 63605:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 2249.712338] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2260.631798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2264.181932] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2266.397267] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2276.339491] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2284.019268] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2295.266545] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2295.270261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2295.286826] Lustre: Skipped 1 previous similar message [ 2295.299196] Lustre: Skipped 4 previous similar messages [ 2298.657812] Lustre: server umount lustre-MDT0000 complete [ 2300.395903] LustreError: 71136:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2300.419156] LustreError: 71136:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 2303.490340] LustreError: 65427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154441 with bad export cookie 11622910440581208524 [ 2303.495781] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2303.498164] LustreError: 65427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2303.811486] Lustre: server umount lustre-MDT0001 complete [ 2312.398243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2316.265637] Lustre: DEBUG MARKER: files compared the same [ 2329.345874] Lustre: server umount lustre-OST0000 complete [ 2346.402000] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 2349.283958] Key type lgssc unregistered [ 2349.534177] LNet: 74415:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2349.542279] LNetError: 74415:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2349.563702] LNet: Removed LNI 192.168.202.128@tcp [ 2350.498573] Key type .llcrypt unregistered [ 2350.505852] Key type ._llcrypt unregistered [ 2374.840775] Key type ._llcrypt registered [ 2374.843216] Key type .llcrypt registered [ 2374.948865] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 2376.255956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2376.402548] alg: No test for adler32 (adler32-zlib) [ 2377.548361] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2377.814862] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 2379.503374] Key type lgssc registered [ 2380.718476] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2390.349081] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2390.376719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2392.255983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2398.490970] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug +malloc all [ 2402.784286] LustreError: 75831:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2402.809016] LustreError: 75831:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2407.906647] LustreError: 75832:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2408.624630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2409.036520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2414.440485] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug +malloc all [ 2423.345572] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2431.294329] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2440.043794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2440.425810] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2449.326827] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug +malloc all [ 2457.389494] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2462.958416] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2473.444790] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2473.450264] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2473.457835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2473.459139] Lustre: Skipped 1 previous similar message [ 2473.468777] Lustre: Skipped 1 previous similar message [ 2476.809334] Lustre: server umount lustre-OST0000 complete [ 2480.616536] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2480.625605] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2480.642088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2483.680055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2483.680232] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2487.145157] Lustre: server umount lustre-MDT0000 complete [ 2488.805420] LustreError: 75827:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2488.852679] LustreError: 75827:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2492.342380] LustreError: 75811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154630 with bad export cookie 3127872540139347146 [ 2492.354308] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2492.363031] LustreError: 75811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2492.999964] Lustre: server umount lustre-MDT0001 complete [ 2506.524972] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 2510.182447] Key type lgssc unregistered [ 2510.540634] LNet: 79385:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2510.555202] LNetError: 79385:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2510.570130] LNet: Removed LNI 192.168.202.128@tcp [ 2511.594779] Key type .llcrypt unregistered [ 2511.598857] Key type ._llcrypt unregistered [ 2539.342388] Key type ._llcrypt registered [ 2539.345170] Key type .llcrypt registered [ 2539.454421] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 2541.642575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2541.679089] alg: No test for adler32 (adler32-zlib) [ 2542.910531] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2543.218914] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 2544.943170] Key type lgssc registered [ 2546.414063] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2554.493413] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2554.511296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2570.143772] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2575.263323] LustreError: 80779:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.128@tcp: failed processing log, type 4: rc = -110 [ 2601.951240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2608.178137] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2616.695345] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2625.370255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2625.550232] Lustre: *** cfs_fail_loc=706, val=0*** [ 2626.026888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2626.117549] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2631.136932] LustreError: 80804:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2631.163118] LustreError: 80804:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2632.070779] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2636.264784] LustreError: 80805:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2636.278309] LustreError: 80805:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2641.380677] LustreError: 80804:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2641.405330] LustreError: 80804:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2641.936574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2642.293331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2648.013325] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2657.290902] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2665.311948] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2668.003813] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2668.005929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2668.021336] Lustre: Skipped 1 previous similar message [ 2668.055535] Lustre: Skipped 1 previous similar message [ 2673.121654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2678.243471] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2678.264382] Lustre: Skipped 2 previous similar messages [ 2682.335154] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2682.687680] Lustre: server umount lustre-OST0000 complete [ 2688.480654] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2688.490108] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2688.510362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2693.602296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2693.617560] Lustre: Skipped 2 previous similar messages [ 2694.286349] Lustre: server umount lustre-MDT0000 complete [ 2698.722252] LustreError: 81870:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2698.749355] LustreError: 81870:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2698.856733] LustreError: 83650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154836 with bad export cookie 10389825288712777761 [ 2698.895051] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2699.299988] Lustre: server umount lustre-MDT0001 complete [ 2711.874493] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 2715.411850] Key type lgssc unregistered [ 2715.647543] LNet: 84402:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2715.655689] LNetError: 84402:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2715.675285] LNet: Removed LNI 192.168.202.128@tcp [ 2716.708565] Key type .llcrypt unregistered [ 2716.710113] Key type ._llcrypt unregistered [ 2743.467828] Key type ._llcrypt registered [ 2743.470491] Key type .llcrypt registered [ 2743.617282] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 2744.794001] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2745.031154] alg: No test for adler32 (adler32-zlib) [ 2746.101917] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2746.441199] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 2748.163305] Key type lgssc registered [ 2749.628181] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2758.138447] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2758.163049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2764.979024] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2773.456994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2774.039721] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2778.903695] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2779.107318] LustreError: 86508:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2779.126339] LustreError: 86508:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2783.202911] LustreError: 86511:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2783.218570] LustreError: 86511:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2788.324045] LustreError: 86512:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2788.337200] LustreError: 86512:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2789.281107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2789.666229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2794.977577] LustreError: 86511:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2795.011165] LustreError: 86511:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2797.826565] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2799.084983] LustreError: 86512:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2799.101965] LustreError: 86512:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2806.240103] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2806.259104] Lustre: MGS: Client ce5cf090-9ecc-44b6-9218-91e53e4d2d5c (at 0@lo) reconnecting [ 2806.304793] LustreError: 86522:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2806.646083] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2806.707516] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2811.128489] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2827.238341] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2827.244878] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2827.258451] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2827.287918] Lustre: Skipped 1 previous similar message [ 2831.984577] Lustre: server umount lustre-OST0000 complete [ 2837.472337] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2837.476188] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2837.489808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2837.489816] Lustre: Skipped 1 previous similar message [ 2837.553738] Lustre: Skipped 1 previous similar message [ 2842.594636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2842.599329] Lustre: Skipped 2 previous similar messages [ 2843.508270] Lustre: server umount lustre-MDT0000 complete [ 2847.718569] LustreError: 86511:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2847.758644] LustreError: 86511:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2850.600193] Lustre: server umount lustre-MDT0001 complete [ 2855.240227] Lustre: server umount MGS complete [ 2870.461386] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 01:43:26 (1788155006) [ 2903.421479] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 2915.291727] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 2960.174382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2965.792796] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2976.811052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2977.091323] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2977.127677] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2977.244450] Lustre: lustre-MDT0001: new disk, initializing [ 2977.333443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2977.360479] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2977.368656] Lustre: 93703:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2977.372780] LustreError: 93703:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2984.456039] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2997.320339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2997.640784] Lustre: lustre-OST0000: new disk, initializing [ 2997.644827] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2997.658084] Lustre: 94469:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2997.726466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3003.928328] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3007.072506] Lustre: 94925:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000006dcc3f28 x1875016295687040/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788155160 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3014.903151] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3022.563676] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3022.577061] Lustre: MGS: Client 7fcdc2a0-5154-4890-9845-288a4d67719e (at 0@lo) reconnecting [ 3022.598909] LustreError: 93689:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3022.651713] Lustre: 92971:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3022.696331] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3022.824421] Lustre: lustre-MDT0000: new disk, initializing [ 3023.048836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3023.110798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3023.143551] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3024.166329] Lustre: 95539:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000004fc38a22 x1875016295694464/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788155178 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3028.080738] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3029.408411] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3033.335725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3033.349352] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3033.500405] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3059.174914] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3059.180829] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3059.196386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3065.503691] Lustre: server umount lustre-OST0000 complete [ 3070.946594] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3070.947146] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3070.963798] LustreError: Skipped 1 previous similar message [ 3070.974334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3070.990687] Lustre: Skipped 2 previous similar messages [ 3071.015936] Lustre: Skipped 3 previous similar messages [ 3080.164088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3080.169841] Lustre: Skipped 3 previous similar messages [ 3084.255140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3084.430852] Lustre: server umount lustre-MDT0000 complete [ 3085.288993] LustreError: 95972:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3085.308664] LustreError: 95972:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3096.157895] Lustre: server umount lustre-MDT0001 complete [ 3100.702326] Lustre: server umount MGS complete [ 3115.912323] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 01:47:31 (1788155251) [ 3135.806581] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 3138.903366] Key type lgssc unregistered [ 3139.216759] LNet: 98039:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3139.225862] LNetError: 98039:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3139.245763] LNet: Removed LNI 192.168.202.128@tcp [ 3140.364710] Key type .llcrypt unregistered [ 3140.369789] Key type ._llcrypt unregistered [ 3161.342089] Key type ._llcrypt registered [ 3161.346082] Key type .llcrypt registered [ 3161.466363] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 3162.717051] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3162.788510] alg: No test for adler32 (adler32-zlib) [ 3164.037511] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 3164.448968] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 3166.119516] Key type lgssc registered [ 3167.508569] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3177.481752] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3177.515628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3179.165582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3184.502768] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3188.707123] LustreError: 99260:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3188.721977] LustreError: 99260:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3193.824079] LustreError: 99261:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3194.760585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3195.159220] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3200.473719] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3211.133455] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3218.523291] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3227.860365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3228.327670] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3235.817556] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3245.044040] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3248.827483] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3264.480983] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3264.485219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3264.512415] Lustre: Skipped 1 previous similar message [ 3264.524703] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3266.924987] Lustre: server umount lustre-OST0000 complete [ 3272.167853] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3272.175250] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3272.199070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3272.211145] Lustre: Skipped 1 previous similar message [ 3274.729078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3274.732839] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3277.563683] Lustre: server umount lustre-MDT0000 complete [ 3279.841972] LustreError: 100244:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3279.857684] LustreError: 100244:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3281.398198] LustreError: 99241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155419 with bad export cookie 4041735946054973786 [ 3281.414311] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3281.791571] Lustre: server umount lustre-MDT0001 complete [ 3293.022627] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 3296.091186] Key type lgssc unregistered [ 3296.417690] LNet: 102820:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3296.437269] LNetError: 102820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3296.452051] LNet: Removed LNI 192.168.202.128@tcp [ 3297.474482] Key type .llcrypt unregistered [ 3297.476645] Key type ._llcrypt unregistered [ 3314.082961] Key type ._llcrypt registered [ 3314.084545] Key type .llcrypt registered [ 3314.162741] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 3315.185137] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3315.209332] alg: No test for adler32 (adler32-zlib) [ 3316.421578] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 3316.771209] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 3318.463177] Key type lgssc registered [ 3319.576941] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3326.149784] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3326.159920] LustreError: 103899:0:(tgt_mount.c:2644:server_fill_super()) cfs_race id 716 sleeping [ 3326.199287] LustreError: 103906:0:(tgt_mount.c:2644:server_fill_super()) cfs_fail_race id 716 waking [ 3326.208817] LustreError: 103899:0:(tgt_mount.c:2644:server_fill_super()) cfs_fail_race id 716 awake: rc=4959 [ 3326.217936] LustreError: 103899:0:(tgt_mount.c:2656:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3326.224353] LustreError: 103899:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3326.227897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3327.828675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3335.088851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3335.586247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3341.247623] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3348.744605] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing lsmod [ 3352.423657] LustreError: 105111:0:(tgt_mount.c:2644:server_fill_super()) cfs_race id 716 sleeping [ 3352.458920] LustreError: 105112:0:(tgt_mount.c:2644:server_fill_super()) cfs_fail_race id 716 waking [ 3352.469296] LustreError: 105111:0:(tgt_mount.c:2644:server_fill_super()) cfs_fail_race id 716 awake: rc=4973 [ 3352.479156] LustreError: 105111:0:(tgt_mount.c:2656:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3352.490988] LustreError: 105111:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3352.521057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3352.921250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3353.995039] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3356.129989] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3356.143184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3356.155160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3359.201382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3359.213334] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3359.230482] Lustre: Skipped 1 previous similar message [ 3360.289249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3360.298094] Lustre: Skipped 4 previous similar messages [ 3362.151498] Lustre: server umount lustre-MDT0000 complete [ 3365.350140] LustreError: 105201:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3365.369068] LustreError: 105201:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3366.919422] LustreError: 103921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155504 with bad export cookie 10104365227356615785 [ 3366.926414] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3366.932127] LustreError: 103921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3368.543351] LustreError: 104373:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000000ed2f236 x1875016893356672/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3368.887298] Lustre: server umount lustre-MDT0001 complete [ 3384.246806] Lustre: server umount lustre-OST0000 complete [ 3395.062583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3395.588748] LustreError: 106177:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3395.656381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3400.696047] LustreError: 106178:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3401.096286] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3405.791952] LustreError: 106177:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3409.193383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3409.740059] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3416.327701] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3424.126654] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3430.294630] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3438.782854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3439.171019] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3441.228107] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3447.251661] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3455.397940] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3461.600835] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3461.605977] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3461.614867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3466.226755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3466.241186] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3466.244065] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3466.246764] Lustre: Skipped 1 previous similar message [ 3467.831656] Lustre: server umount lustre-OST0000 complete [ 3476.449517] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3476.452264] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3476.480423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3476.499870] Lustre: Skipped 1 previous similar message [ 3486.175164] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3486.423788] Lustre: server umount lustre-MDT0000 complete [ 3486.692760] LustreError: 107100:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3486.701957] LustreError: 107100:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3490.240748] LustreError: 106159:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155628 with bad export cookie 10104365227356616716 [ 3490.257420] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3490.755893] Lustre: server umount lustre-MDT0001 complete [ 3503.114908] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 3506.491582] Key type lgssc unregistered [ 3506.786507] LNet: 109713:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3506.793941] LNetError: 109713:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3506.809168] LNet: Removed LNI 192.168.202.128@tcp [ 3507.730634] Key type .llcrypt unregistered [ 3507.736280] Key type ._llcrypt unregistered [ 3531.285257] Key type ._llcrypt registered [ 3531.289938] Key type .llcrypt registered [ 3531.427877] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 3532.393702] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3532.422041] alg: No test for adler32 (adler32-zlib) [ 3533.806332] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 3534.151839] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 3535.857648] Key type lgssc registered [ 3536.724228] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3544.758341] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3544.770630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3546.189817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3550.375791] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3556.321733] LustreError: 111132:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3556.341223] LustreError: 111132:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3559.681070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3559.984285] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3564.234465] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3574.873811] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3583.405691] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3592.323599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3592.870887] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3599.632915] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3608.895535] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3616.493671] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3625.792135] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3631.142275] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3637.217364] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3637.223890] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3637.238211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3637.268702] Lustre: Skipped 1 previous similar message [ 3642.359237] Lustre: server umount lustre-OST0000 complete [ 3646.944523] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3646.954250] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3646.964539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3646.977364] Lustre: Skipped 1 previous similar message [ 3652.583169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3652.600144] Lustre: Skipped 3 previous similar messages [ 3652.746829] Lustre: server umount lustre-MDT0000 complete [ 3657.202128] LustreError: 111113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155795 with bad export cookie 16821894482718829379 [ 3657.206202] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3657.215720] LustreError: 111113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3657.594236] Lustre: server umount lustre-MDT0001 complete [ 3671.962813] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 3675.374721] Key type lgssc unregistered [ 3675.724099] LNet: 114791:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3675.743390] LNetError: 114791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3675.765139] LNet: Removed LNI 192.168.202.128@tcp [ 3677.033668] Key type .llcrypt unregistered [ 3677.041193] Key type ._llcrypt unregistered [ 3699.332433] Key type ._llcrypt registered [ 3699.337625] Key type .llcrypt registered [ 3699.477444] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 3701.180899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3701.239666] alg: No test for adler32 (adler32-zlib) [ 3702.390268] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 3702.662310] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 3704.383991] Key type lgssc registered [ 3706.034290] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3715.903673] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3715.918450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3717.625914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3723.271849] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3727.845813] LustreError: 116013:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3727.872482] LustreError: 116013:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3732.963080] LustreError: 116013:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3734.216252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3734.640793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3740.086222] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3749.913418] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3757.476709] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3766.012371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3766.446444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3773.646932] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3782.505134] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3787.971242] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3798.506464] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3798.510495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3798.511187] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3798.523028] Lustre: Skipped 1 previous similar message [ 3798.551502] Lustre: Skipped 1 previous similar message [ 3799.273672] Lustre: server umount lustre-OST0000 complete [ 3803.618146] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3803.631831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3803.635721] Lustre: Skipped 1 previous similar message [ 3803.648440] Lustre: Skipped 1 previous similar message [ 3808.738400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3808.741892] Lustre: Skipped 1 previous similar message [ 3809.813205] Lustre: server umount lustre-MDT0000 complete [ 3813.856821] LustreError: 116008:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3813.869142] LustreError: 116008:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3813.915573] LustreError: 115994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155951 with bad export cookie 9968929578719019664 [ 3813.932687] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3814.316383] Lustre: server umount lustre-MDT0001 complete [ 3826.677157] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 3830.256500] Key type lgssc unregistered [ 3830.699724] LNet: 119565:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3830.703918] LNetError: 119565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3830.725887] LNet: Removed LNI 192.168.202.128@tcp [ 3831.687925] Key type .llcrypt unregistered [ 3831.689550] Key type ._llcrypt unregistered [ 3862.263461] Key type ._llcrypt registered [ 3862.266692] Key type .llcrypt registered [ 3862.437850] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 3864.609362] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3864.762089] alg: No test for adler32 (adler32-zlib) [ 3865.956088] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 3866.161787] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 3867.847744] Key type lgssc registered [ 3869.628817] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3880.097350] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3880.112744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3881.581723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3887.485346] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3891.174368] LustreError: 120987:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3891.205586] LustreError: 120987:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3896.303253] LustreError: 120986:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3898.024175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3898.413548] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3903.349697] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3912.481390] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3920.196764] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3928.300324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3928.751714] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3936.823769] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3946.336305] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3951.294720] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3958.707220] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3962.441126] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3962.444122] Lustre: Skipped 2 previous similar messages [ 3966.605945] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3966.648283] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3967.798762] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3967.807177] Lustre: Skipped 2 previous similar messages [ 3973.780515] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3973.800177] Lustre: Skipped 1 previous similar message [ 3979.165619] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.28@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3979.173815] Lustre: Skipped 2 previous similar messages [ 3981.705874] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.28@tcp 192.168.0.[2,10]@tcp [ 3992.251840] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.28/24@tcp in log lustre-MDT0000 [ 3992.257912] Lustre: Skipped 2 previous similar messages [ 4001.361569] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.28/24@tcp [ 4001.374643] Lustre: Skipped 1 previous similar message [ 4007.905711] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4007.913988] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4007.923806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4007.954966] Lustre: Skipped 1 previous similar message [ 4009.429059] Lustre: server umount lustre-OST0000 complete [ 4016.099107] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4016.109982] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4016.126797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4016.136305] Lustre: Skipped 1 previous similar message [ 4018.147558] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4018.148376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4018.185232] Lustre: Skipped 1 previous similar message [ 4023.268136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4023.272757] Lustre: Skipped 1 previous similar message [ 4027.366800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4027.385364] Lustre: Skipped 1 previous similar message [ 4028.383123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4028.619296] Lustre: server umount lustre-MDT0000 complete [ 4032.498114] LustreError: 120986:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4032.534764] LustreError: 120986:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4032.919671] LustreError: 120967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156170 with bad export cookie 1825319321331100611 [ 4032.939371] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4032.942935] LustreError: 120967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4033.308145] Lustre: server umount lustre-MDT0001 complete [ 4046.224233] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 4050.038068] Key type lgssc unregistered [ 4050.552537] LNet: 125950:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4050.558133] LNetError: 125950:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4051.623812] LNet: Removed LNI 192.168.202.128@tcp [ 4052.769350] Key type .llcrypt unregistered [ 4052.771495] Key type ._llcrypt unregistered [ 4090.267176] Key type ._llcrypt registered [ 4090.280614] Key type .llcrypt registered [ 4090.479244] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 4091.735987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4091.884427] alg: No test for adler32 (adler32-zlib) [ 4093.049581] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 4093.387179] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 4095.159287] Key type lgssc registered [ 4096.604429] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4106.297369] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4106.336470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4107.719545] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4107.740056] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.28/24@tcp [ 4107.786304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4112.972404] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4117.993852] LustreError: 127515:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4118.014179] LustreError: 127515:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4123.104565] LustreError: 127516:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4123.390481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4123.828608] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4123.852951] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.28/24@tcp [ 4123.922116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4128.922620] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4137.595509] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4143.778963] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4151.254296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4151.710580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4152.795225] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 4159.256461] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4167.900307] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4177.900567] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4177.911455] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4177.938469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4179.429709] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4179.433281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4179.438365] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4179.460202] Lustre: Skipped 1 previous similar message [ 4183.925154] Lustre: server umount lustre-OST0000 complete [ 4189.673996] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4189.689267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4194.084590] Lustre: server umount lustre-MDT0000 complete [ 4194.789080] LustreError: 129649:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4194.810017] LustreError: 129649:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4198.699440] LustreError: 127496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156336 with bad export cookie 2536594894048856001 [ 4198.700129] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4198.708335] LustreError: 127496:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4199.193929] Lustre: server umount lustre-MDT0001 complete [ 4212.549203] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 4216.001592] Key type lgssc unregistered [ 4216.331632] LNet: 131105:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4216.341603] LNetError: 131105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4217.381066] LNet: Removed LNI 192.168.202.128@tcp [ 4218.647654] Key type .llcrypt unregistered [ 4218.653784] Key type ._llcrypt unregistered [ 4248.161967] Key type ._llcrypt registered [ 4248.163838] Key type .llcrypt registered [ 4248.294777] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 4265.859603] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 4267.233810] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4267.370831] alg: No test for adler32 (adler32-zlib) [ 4268.602609] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 4268.870540] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 4270.663337] Key type lgssc registered [ 4271.754332] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4311.274431] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4311.311149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4312.735895] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4312.761440] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4312.889652] Lustre: lustre-MDT0000: new disk, initializing [ 4313.055154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4313.101614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4318.704460] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4332.126839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4332.299806] Lustre: 134506:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4332.444717] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4332.454156] Lustre: Skipped 1 previous similar message [ 4332.609430] Lustre: lustre-MDT0001: new disk, initializing [ 4332.761573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4332.819939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4332.842401] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4337.604583] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4350.287480] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4357.219851] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4365.506498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4366.003967] Lustre: lustre-OST0000: new disk, initializing [ 4366.010132] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4366.025898] Lustre: 136479:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4366.130189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4373.096507] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4374.639997] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4374.652783] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4374.812553] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4384.966636] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4390.780378] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4391.117441] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4396.747976] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4397.031513] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4400.610257] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4400.613556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4400.613641] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4400.635384] Lustre: Skipped 1 previous similar message [ 4400.661796] Lustre: Skipped 1 previous similar message [ 4405.719589] Lustre: server umount lustre-OST0000 complete [ 4414.947742] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4414.961700] LustreError: Skipped 1 previous similar message [ 4414.976574] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4415.002230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4417.267928] Lustre: server umount lustre-MDT0000 complete [ 4421.731967] LustreError: 134500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156559 with bad export cookie 16781420320282146479 [ 4421.733125] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4421.754313] LustreError: 134500:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4422.253988] Lustre: server umount lustre-MDT0001 complete [ 4432.525357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4432.995530] LustreError: 138695:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4433.030521] LustreError: 138695:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4433.100918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4437.074994] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4438.497479] LustreError: 138696:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4438.519658] LustreError: 138696:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4442.596457] LustreError: 138695:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4446.141724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4446.490377] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4451.811760] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4460.410823] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4467.465842] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4475.035371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4475.501519] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4482.587780] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4492.810320] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4499.162950] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4499.340794] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4504.708365] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4505.047218] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4515.615683] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4515.792405] Lustre: lustre-OST0001: new disk, initializing [ 4515.795569] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4515.800984] Lustre: 142069:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4515.898629] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4522.655453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4522.663862] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4522.773825] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4523.366898] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4534.681233] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4534.938631] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4541.430472] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4541.731049] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4547.522830] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff978c104e2000.ost_server_uuid 50 [ 4549.192325] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff978c104e2000.ost_server_uuid in IDLE state after 0 sec [ 4559.784195] Lustre: server umount lustre-OST0001 complete [ 4563.936020] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4563.945037] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4563.963363] Lustre: Skipped 2 previous similar messages [ 4563.966903] LustreError: 140520:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4563.986281] LustreError: 140520:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4564.965919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4564.975316] Lustre: Skipped 2 previous similar messages [ 4569.056528] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4569.066592] LustreError: Skipped 2 previous similar messages [ 4569.068164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4569.068633] LustreError: 140520:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4569.074018] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4569.074027] Lustre: Skipped 1 previous similar message [ 4569.129242] Lustre: Skipped 1 previous similar message [ 4570.664554] Lustre: server umount lustre-OST0000 complete [ 4579.296451] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4579.300171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4579.335426] Lustre: Skipped 1 previous similar message [ 4579.348685] Lustre: Skipped 1 previous similar message [ 4581.707279] Lustre: server umount lustre-MDT0000 complete [ 4584.417382] LustreError: 138690:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4584.473955] LustreError: 138690:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4587.269144] LustreError: 142068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156725 with bad export cookie 16781420320282147361 [ 4587.289689] LustreError: 142068:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4587.291571] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4588.066315] Lustre: server umount lustre-MDT0001 complete [ 4616.092310] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 4619.970141] Key type lgssc unregistered [ 4620.484688] LNet: 145076:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4620.500327] LNetError: 145076:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4620.523529] LNet: Removed LNI 192.168.202.128@tcp [ 4621.790812] Key type .llcrypt unregistered [ 4621.792085] Key type ._llcrypt unregistered [ 4667.822790] Key type ._llcrypt registered [ 4667.825332] Key type .llcrypt registered [ 4667.935656] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 4683.366405] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 4684.886681] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4684.931090] alg: No test for adler32 (adler32-zlib) [ 4685.998731] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 4686.337185] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 4687.992716] Key type lgssc registered [ 4689.379771] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4725.092435] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4725.115923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4726.338424] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4726.368337] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4726.485286] Lustre: lustre-MDT0000: new disk, initializing [ 4726.548148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4726.561645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4730.949022] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4742.438510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4742.562952] Lustre: 149230:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4742.611265] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4742.614345] Lustre: Skipped 1 previous similar message [ 4742.679659] Lustre: lustre-MDT0001: new disk, initializing [ 4742.746578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4742.786608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4742.801467] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4747.246149] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4757.169501] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4763.731914] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4771.055544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4771.439293] Lustre: lustre-OST0000: new disk, initializing [ 4771.444918] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4771.449617] Lustre: 151201:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4771.525582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4777.058640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4777.066145] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4777.244980] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4779.406584] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4792.052065] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4797.344859] Lustre: Failing over lustre-OST0000 [ 4797.823927] Lustre: server umount lustre-OST0000 complete [ 4813.284409] Lustre: 147455:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788156935/real 1788156935] req@000000007c3ae184 x1875018329917184/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788156951 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4813.285327] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4813.306340] Lustre: 147455:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4813.329347] Lustre: Skipped 1 previous similar message [ 4814.371125] Lustre: 147458:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788156936/real 1788156936] req@0000000004ec2f8c x1875018329917952/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788156952 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4818.400366] Lustre: 147455:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788156940/real 1788156940] req@00000000a1cba1c2 x1875018329918848/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788156956 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4818.475637] Lustre: 147455:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4819.420660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4819.738918] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4819.757840] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4823.846420] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4824.863117] Lustre: 147458:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788156946/real 1788156946] req@00000000b6a5ad96 x1875018329920000/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788156962 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4825.612433] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4826.935625] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4826.937274] Lustre: lustre-OST0000: Recovery over after 0:03, of 3 clients 3 recovered and 0 were evicted. [ 4830.166329] Lustre: 147457:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788156951/real 1788156951] req@00000000bc13939f x1875018329921152/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788156967 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4830.174285] Lustre: Failing over lustre-MDT0000 [ 4830.197660] Lustre: 147457:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4830.694634] Lustre: server umount lustre-MDT0000 complete [ 4834.280120] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4834.307180] Lustre: Skipped 2 previous similar messages [ 4837.086767] LustreError: 151671:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4837.104310] LustreError: 151671:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4839.398678] LustreError: 149235:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4839.417581] LustreError: 149235:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4842.204284] LustreError: 149239:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4844.513953] LustreError: 149243:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4844.533584] LustreError: 149243:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4848.516389] LustreError: 151456:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4848.555620] LustreError: 151456:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4850.613543] Lustre: 147455:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788156972/real 1788156972] req@000000009513a5e2 x1875018329931264/t0(0) o400->MGC192.168.202.128@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788156988 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4850.702231] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4855.707206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4858.593298] LustreError: 149238:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4858.616474] LustreError: 149238:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 4861.985308] LustreError: 147454:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000006867b906 x1875018329937408/t0(0) o250->MGC192.168.202.128@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4862.294478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4862.363981] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4864.735246] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4867.596746] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4867.617805] Lustre: Skipped 1 previous similar message [ 4867.659884] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4867.733318] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4869.120608] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4878.307410] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4878.307590] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4878.310147] LustreError: Skipped 1 previous similar message [ 4878.311463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4878.327633] Lustre: Skipped 1 previous similar message [ 4882.923742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4882.941272] Lustre: Skipped 2 previous similar messages [ 4884.360808] Lustre: server umount lustre-OST0000 complete [ 4888.544256] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4888.554802] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4888.595825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4893.154949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4893.155953] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4894.607215] Lustre: server umount lustre-MDT0000 complete [ 4898.273954] LustreError: 149243:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4898.322624] LustreError: 149243:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 4899.365089] LustreError: 149223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157037 with bad export cookie 12732765248240721876 [ 4899.376020] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4899.379640] LustreError: 149223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4899.941877] Lustre: server umount lustre-MDT0001 complete [ 4914.243529] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 4917.464774] Key type lgssc unregistered [ 4917.858335] LNet: 154860:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4917.873341] LNetError: 154860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4917.911841] LNet: Removed LNI 192.168.202.128@tcp [ 4919.323304] Key type .llcrypt unregistered [ 4919.327198] Key type ._llcrypt unregistered [ 4958.316185] Key type ._llcrypt registered [ 4958.318633] Key type .llcrypt registered [ 4958.472260] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 4960.082784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4960.127567] alg: No test for adler32 (adler32-zlib) [ 4961.282472] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 4961.601669] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 4963.255320] Key type lgssc registered [ 4964.631961] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4973.140166] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4973.158853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4974.687434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4980.692819] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4984.806476] LustreError: 156384:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4984.832865] LustreError: 156384:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4989.922861] LustreError: 156383:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4992.146558] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4995.042392] LustreError: 156384:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4997.294670] LustreError: 156412:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4997.298872] LustreError: 156412:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4997.303576] LustreError: 156412:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 23, retries 0, failed: rc = -5 [ 4997.508271] Lustre: server umount lustre-MDT0000 complete [ 5018.209384] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 5034.508533] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 5076.789056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5077.102116] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5077.121405] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5077.174699] Lustre: lustre-MDT0000: new disk, initializing [ 5077.235673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5077.248272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5082.038083] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5094.589402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5094.704952] Lustre: 159740:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5094.779791] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5094.784327] Lustre: Skipped 1 previous similar message [ 5094.863708] Lustre: lustre-MDT0001: new disk, initializing [ 5094.971044] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5095.024253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5095.048125] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5100.157778] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5112.675407] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5120.105504] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5129.448310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5129.745499] Lustre: lustre-OST0000: new disk, initializing [ 5129.748274] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5129.753715] Lustre: 161710:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5129.852778] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5136.985451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5136.999030] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5137.119764] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5137.621712] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5150.980651] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5155.706534] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5158.704458] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 02:21:33 (1788157293) [ 5444.581963] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5444.584221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5444.603651] Lustre: Skipped 1 previous similar message [ 5444.626665] Lustre: Skipped 2 previous similar messages [ 5449.697214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5449.706121] Lustre: Skipped 2 previous similar messages [ 5450.178341] Lustre: server umount lustre-MDT0000 complete [ 5454.126603] LustreError: 159731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157592 with bad export cookie 17992531157910398121 [ 5454.127774] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5454.142034] LustreError: 159731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5454.815746] LustreError: 159747:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5454.821239] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5454.836547] Lustre: Skipped 1 previous similar message [ 5454.840424] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5459.938892] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5459.939433] LustreError: 161718:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5460.511494] Lustre: server umount lustre-MDT0001 complete [ 5472.285273] Lustre: server umount lustre-OST0000 complete [ 5484.130740] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 5501.045983] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 5547.289593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5547.604196] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5547.670228] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5547.789597] Lustre: lustre-MDT0000: new disk, initializing [ 5547.917186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5547.939542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5554.339160] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5566.621566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5566.781774] Lustre: 166407:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5566.840827] LustreError: 166417:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5566.871170] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5566.877660] Lustre: Skipped 1 previous similar message [ 5566.956421] Lustre: lustre-MDT0001: new disk, initializing [ 5567.046970] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5567.092807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5567.111578] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5573.220890] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5585.467868] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5593.237727] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5601.154918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5601.444593] Lustre: lustre-OST0000: new disk, initializing [ 5601.455176] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5601.466382] Lustre: 168375:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5601.619739] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5608.689702] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5609.607086] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5609.630371] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5609.794575] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5620.120257] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5625.885582] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5626.105101] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5631.453301] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5631.661098] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5635.559540] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5635.560693] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5635.569366] LustreError: Skipped 1 previous similar message [ 5635.575361] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5635.582768] Lustre: Skipped 1 previous similar message [ 5635.610574] Lustre: Skipped 1 previous similar message [ 5645.804867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5645.812849] Lustre: Skipped 3 previous similar messages [ 5648.351140] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5648.552180] Lustre: server umount lustre-OST0000 complete [ 5653.986389] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5653.998639] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5659.984378] Lustre: server umount lustre-MDT0000 complete [ 5661.176502] LustreError: 166413:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5661.203287] LustreError: 166413:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5665.165956] LustreError: 169917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157803 with bad export cookie 17992531157910556958 [ 5665.178056] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5665.184056] LustreError: 169917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5665.913701] Lustre: server umount lustre-MDT0001 complete [ 5677.486456] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:30:13 (1788157813) [ 5700.844602] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 5713.286436] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 5759.255566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5759.526521] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5759.567448] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5759.626443] Lustre: lustre-MDT0000: new disk, initializing [ 5759.705325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5759.719486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5764.662973] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5779.182744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5779.412536] Lustre: 173135:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5779.815049] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5785.792652] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5797.439535] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5803.761281] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5811.965830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5812.492549] Lustre: lustre-OST0000: new disk, initializing [ 5812.495870] Lustre: Skipped 1 previous similar message [ 5812.500734] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5812.506597] Lustre: Skipped 2 previous similar messages [ 5812.523726] Lustre: 175114:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5819.616556] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5820.471877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5820.484938] Lustre: Skipped 1 previous similar message [ 5820.495136] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5820.668900] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5830.244670] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5841.376248] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5841.379990] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5841.388351] LustreError: Skipped 1 previous similar message [ 5841.388829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5841.388832] Lustre: Skipped 3 previous similar messages [ 5841.417249] Lustre: Skipped 2 previous similar messages [ 5844.697662] Lustre: server umount lustre-OST0000 complete [ 5851.616052] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5851.618904] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5851.643627] Lustre: Skipped 1 previous similar message [ 5855.933327] Lustre: server umount lustre-MDT0000 complete [ 5856.737893] LustreError: 175701:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5856.745206] LustreError: 175701:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5860.172026] LustreError: 173127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157998 with bad export cookie 17992531157910557840 [ 5860.176433] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5860.186586] LustreError: 173127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5860.569856] Lustre: server umount lustre-MDT0001 complete [ 5872.256823] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:33:27 (1788158007) [ 5897.408600] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 5909.231877] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 5953.552560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5953.838968] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5953.873692] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5954.007739] Lustre: lustre-MDT0000: new disk, initializing [ 5954.126468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5954.131871] Lustre: Skipped 2 previous similar messages [ 5954.164457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5960.306889] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5973.034185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5973.202992] Lustre: 179440:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5973.469087] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5978.745622] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5990.343262] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5996.839952] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6005.044342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6005.480582] Lustre: 181418:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6011.950651] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6012.028339] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6012.200132] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6024.410986] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6037.993447] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6037.994779] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6038.010308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6038.010319] Lustre: Skipped 3 previous similar messages [ 6038.013568] Lustre: Skipped 1 previous similar message [ 6039.620850] Lustre: server umount lustre-OST0000 complete [ 6050.474799] Lustre: server umount lustre-MDT0000 complete [ 6053.345477] LustreError: 179448:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6053.362239] LustreError: 179448:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6055.089751] LustreError: 180240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158192 with bad export cookie 17992531157910558925 [ 6055.103417] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6055.108722] LustreError: 180240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6055.795819] Lustre: server umount lustre-MDT0001 complete [ 6069.155448] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 6072.050780] Key type lgssc unregistered [ 6072.431503] LNet: 183273:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6072.449957] LNetError: 183273:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6072.473682] LNet: Removed LNI 192.168.202.128@tcp [ 6073.453117] Key type .llcrypt unregistered [ 6073.455950] Key type ._llcrypt unregistered [ 6100.290538] Key type ._llcrypt registered [ 6100.293337] Key type .llcrypt registered [ 6100.409566] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 6101.418922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6101.438212] alg: No test for adler32 (adler32-zlib) [ 6102.662281] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 6103.013719] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 6104.818815] Key type lgssc registered [ 6106.765458] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6118.075514] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6118.114679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6119.652451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6126.040947] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6129.147934] LustreError: 184692:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6129.173917] LustreError: 184692:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6134.240861] LustreError: 184693:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6135.440606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6135.671930] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6139.541875] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6147.428328] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6154.155441] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6161.710765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6162.189041] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6164.279883] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6170.046117] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6178.492754] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6200.288315] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6200.293866] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6200.306898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6201.318222] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6201.335656] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6201.337154] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6201.366968] Lustre: Skipped 1 previous similar message [ 6204.173427] Lustre: server umount lustre-OST0000 complete [ 6211.561320] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6211.566574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6211.579718] Lustre: Skipped 1 previous similar message [ 6211.591390] Lustre: Skipped 1 previous similar message [ 6216.672439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6216.676142] Lustre: Skipped 1 previous similar message [ 6221.792971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6221.800516] Lustre: Skipped 1 previous similar message [ 6222.303140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6222.467413] Lustre: server umount lustre-MDT0000 complete [ 6226.414262] LustreError: 184674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158364 with bad export cookie 14179989574631298467 [ 6226.415977] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6226.434321] LustreError: 184674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6226.910741] Lustre: server umount lustre-MDT0001 complete [ 6237.097291] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 6239.706800] Key type lgssc unregistered [ 6240.057518] LNet: 188250:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6240.064125] LNetError: 188250:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6240.080828] LNet: Removed LNI 192.168.202.128@tcp [ 6240.936110] Key type .llcrypt unregistered [ 6240.937499] Key type ._llcrypt unregistered [ 6261.913816] Key type ._llcrypt registered [ 6261.915259] Key type .llcrypt registered [ 6262.008285] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 6262.920198] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6262.959514] alg: No test for adler32 (adler32-zlib) [ 6263.997944] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 6264.285587] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 6265.967185] Key type lgssc registered [ 6267.162989] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6275.207276] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6275.232639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6276.963749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6281.908349] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6286.308191] LustreError: 189646:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6286.324305] LustreError: 189646:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6291.430943] LustreError: 189647:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6292.894739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6293.261406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6298.584563] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6307.644622] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6313.567042] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6320.430629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6320.818197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6321.841546] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6326.915328] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6335.040900] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6347.508221] Lustre: lustre-OST0000: Not available for connect from 192.168.202.28@tcp (stopping) [ 6347.744721] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6347.757102] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6348.768462] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6348.782111] Lustre: Skipped 1 previous similar message [ 6348.783066] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6348.797953] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6351.158739] Lustre: server umount lustre-OST0000 complete [ 6358.994798] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff978c05fe9000.ost_server_uuid 50 [ 6360.796831] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6375.392853] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6375.401717] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6375.422172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6375.424970] Lustre: Skipped 1 previous similar message [ 6378.464221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6378.466792] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6380.600763] Lustre: server umount lustre-MDT0000 complete [ 6383.594055] LustreError: 189641:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6383.627350] LustreError: 189641:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6384.075326] LustreError: 189626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158521 with bad export cookie 13102542140082399935 [ 6384.089955] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6384.803655] Lustre: server umount lustre-MDT0001 complete [ 6393.323402] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 02:42:09 (1788158529) [ 6400.525754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6400.936679] LustreError: 193563:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6401.008267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6405.688710] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6406.117033] LustreError: 193564:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6415.335111] LustreError: 193564:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6415.346602] LustreError: 193564:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6415.369928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6415.864638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6422.201170] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6432.411568] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6439.694445] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6449.094649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6449.459279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6457.938531] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6466.721365] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6470.966293] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6474.428871] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6474.659404] Lustre: lustre-OST0001: new disk, initializing [ 6474.666777] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6474.676093] Lustre: 196408:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6474.847457] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6482.064774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6482.078819] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6482.250258] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6482.251817] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6492.342484] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6496.736402] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6496.737678] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6496.746876] LustreError: Skipped 1 previous similar message [ 6496.749984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6496.762593] Lustre: Skipped 1 previous similar message [ 6496.786849] Lustre: Skipped 2 previous similar messages [ 6501.580477] Lustre: server umount lustre-OST0000 complete [ 6502.901654] LustreError: 195387:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6502.914302] LustreError: 195387:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6510.072482] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6510.414048] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6515.835971] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6516.163661] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6533.616959] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6533.626522] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6533.644861] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6533.647479] Lustre: Skipped 3 previous similar messages [ 6536.647880] Lustre: server umount lustre-OST0001 complete [ 6548.012214] Lustre: server umount lustre-MDT0000 complete [ 6548.971162] LustreError: 193558:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6549.031304] LustreError: 193558:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [ 6553.083709] LustreError: 193544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158690 with bad export cookie 13102542140082401006 [ 6553.089260] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6553.093836] LustreError: 193544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6553.898371] Lustre: server umount lustre-MDT0001 complete [ 6581.549727] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 02:45:17 (1788158717) [ 6588.702644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6588.984987] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6589.019467] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6589.169197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6594.941409] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6604.796564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6604.945851] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6604.989477] Lustre: 199848:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6610.383796] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6620.301045] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6627.429585] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6635.892890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6636.187567] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6642.941700] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6644.735684] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6654.131704] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6662.695187] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6662.825628] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6663.304745] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6663.314343] Lustre: Skipped 2 previous similar messages [ 6670.919612] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6679.502554] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6684.129074] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6684.135417] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6684.135427] Lustre: Skipped 3 previous similar messages [ 6684.136986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6684.153656] LustreError: Skipped 2 previous similar messages [ 6684.178967] Lustre: Skipped 3 previous similar messages [ 6688.685187] Lustre: server umount lustre-OST0000 complete [ 6690.541634] LustreError: 201697:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6707.183925] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6710.279438] Lustre: server umount lustre-OST0001 complete [ 6717.410957] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6717.412736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6717.440252] Lustre: Skipped 4 previous similar messages [ 6717.479411] Lustre: Skipped 6 previous similar messages [ 6729.183202] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6729.742242] Lustre: server umount lustre-MDT0000 complete [ 6734.236832] LustreError: 202734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158872 with bad export cookie 13102542140082402413 [ 6734.249649] LustreError: 202734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6734.259325] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6734.741826] Lustre: server umount lustre-MDT0001 complete [ 6761.103296] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 02:48:16 (1788158896) [ 6782.996599] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 6795.191744] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 6836.317653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6836.596159] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6836.634364] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6836.709532] Lustre: lustre-MDT0000: new disk, initializing [ 6836.852054] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6836.877462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6841.777657] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6852.776960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6852.882656] Lustre: 207949:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6852.935234] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6852.941202] Lustre: Skipped 1 previous similar message [ 6853.031885] Lustre: lustre-MDT0001: new disk, initializing [ 6853.211321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6853.222730] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6858.520765] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6868.842362] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6874.223701] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6881.896914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6882.175528] Lustre: lustre-OST0000: new disk, initializing [ 6882.180084] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6882.188474] Lustre: 209918:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6888.828385] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6891.606083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6891.623846] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6891.736532] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6899.613962] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6904.785195] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6905.038949] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6909.765973] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6910.073331] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6917.628319] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6917.636017] LustreError: Skipped 1 previous similar message [ 6917.639306] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6917.649673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6917.654887] Lustre: Skipped 4 previous similar messages [ 6918.701454] Lustre: server umount lustre-OST0000 complete [ 6929.698286] Lustre: server umount lustre-MDT0000 complete [ 6932.970092] LustreError: 207955:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6933.003312] LustreError: 207955:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 6934.157917] LustreError: 207942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159072 with bad export cookie 13102542140082403974 [ 6934.166258] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6934.173021] LustreError: 207942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6934.568742] Lustre: server umount lustre-MDT0001 complete [ 6944.404166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6950.115932] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6959.497566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6964.858786] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6974.925429] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6981.584444] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6989.253990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6996.656485] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7004.982353] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7010.200464] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7010.513695] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7015.255572] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7015.528723] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7020.526711] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7020.534964] LustreError: Skipped 2 previous similar messages [ 7024.060937] Lustre: server umount lustre-OST0000 complete [ 7032.524982] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7038.172721] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7068.290361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7075.830890] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7083.189271] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7086.057713] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7086.079935] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7086.081180] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7086.192380] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 7091.168280] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7091.183831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7091.188678] Lustre: Skipped 6 previous similar messages [ 7091.201758] Lustre: Skipped 6 previous similar messages [ 7096.800475] Lustre: server umount lustre-OST0000 complete [ 7107.606953] Lustre: server umount lustre-MDT0000 complete [ 7112.797654] LustreError: 212118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159250 with bad export cookie 13102542140082404856 [ 7112.829509] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7113.561178] Lustre: server umount lustre-MDT0001 complete [ 7125.452231] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 02:54:21 (1788159261) [ 7134.085576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7134.936649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7134.948779] Lustre: Skipped 6 previous similar messages [ 7140.809902] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7151.417435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7156.693335] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7165.604420] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7171.846373] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7179.797814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7186.526964] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7195.269878] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7200.959808] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7205.442636] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 7210.430875] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7210.750850] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7218.618484] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7218.874525] Lustre: lustre-OST0001: new disk, initializing [ 7218.882984] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7218.896042] Lustre: 221506:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7228.766655] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7229.022496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7229.033193] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7229.194388] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7239.757754] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7244.523528] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7244.714550] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7249.218689] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7249.438378] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7255.014507] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7255.024053] LustreError: Skipped 2 previous similar messages [ 7257.863935] Lustre: server umount lustre-OST0001 complete [ 7260.152119] LustreError: 219928:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7260.177158] LustreError: 219928:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 7265.247688] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7265.443418] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7270.616896] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7270.852899] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7301.309400] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7306.729023] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7306.768054] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7306.777235] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7306.799087] Lustre: Skipped 2 previous similar messages [ 7309.236534] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7317.994843] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7320.201309] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7328.863983] Lustre: server umount lustre-OST0001 complete [ 7347.684573] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7347.718424] Lustre: Skipped 6 previous similar messages [ 7347.733369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7347.740586] Lustre: Skipped 10 previous similar messages [ 7360.991142] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7361.160179] Lustre: server umount lustre-OST0000 complete [ 7372.423890] Lustre: server umount lustre-MDT0000 complete [ 7376.888202] LustreError: 218087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159514 with bad export cookie 13102542140082405815 [ 7376.894304] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7376.898547] LustreError: 218087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 7383.394575] Lustre: server umount lustre-MDT0001 complete [ 7415.193602] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 02:59:10 (1788159550) [ 7422.833446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7423.307861] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7423.392387] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7429.441790] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7439.749703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7439.834431] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7439.895910] Lustre: 226755:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7445.514494] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7454.223299] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7461.397863] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7469.636954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7469.912515] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7477.919988] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7486.709561] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7497.205970] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7497.271790] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7505.183810] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7514.257330] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7516.575114] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7522.608793] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7522.912412] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7528.404533] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7528.607755] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7535.236257] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff978c05328800.ost_server_uuid 50 [ 7536.860191] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff978c05328800.ost_server_uuid in IDLE state after 0 sec [ 7537.858642] Lustre: Permanently deactivating lustre-OST0001 [ 7537.883308] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7541.901368] Lustre: Permanently reactivating lustre-OST0001 [ 7541.942111] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7541.958150] Lustre: Skipped 2 previous similar messages [ 7546.853053] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7546.861868] LustreError: Skipped 1 previous similar message [ 7552.407192] Lustre: server umount lustre-OST0001 complete [ 7562.693440] Lustre: server umount lustre-OST0000 complete [ 7573.590191] Lustre: server umount lustre-MDT0000 complete [ 7577.626126] LustreError: 227498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159715 with bad export cookie 13102542140082407110 [ 7577.628430] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7577.643944] LustreError: 227498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7578.391754] Lustre: server umount lustre-MDT0001 complete [ 7606.645464] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 03:02:22 (1788159742) [ 7615.500105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7615.742283] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7615.766857] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7615.778695] Lustre: Skipped 2 previous similar messages [ 7621.253616] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7630.365561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7630.477728] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7630.552326] Lustre: 233464:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7635.181734] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7642.049673] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7647.508499] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7654.203374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7654.410522] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7654.436957] Lustre: Permanently deactivating lustre-OST0000 [ 7654.455232] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7654.625593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7654.633094] Lustre: Skipped 10 previous similar messages [ 7660.103051] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7660.813902] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7667.981043] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7674.670637] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7674.747596] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7681.081870] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7683.249850] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7688.641949] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7690.611264] Lustre: Permanently reactivating lustre-OST0000 [ 7690.640983] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7690.649281] Lustre: Skipped 2 previous similar messages [ 7691.515245] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7691.529211] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7691.542072] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7691.594669] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7691.615186] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7712.118415] Lustre: server umount lustre-OST0001 complete [ 7722.553197] Lustre: server umount lustre-OST0000 complete [ 7732.651429] Lustre: server umount lustre-MDT0000 complete [ 7736.054101] LustreError: 233454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159873 with bad export cookie 13102542140082408755 [ 7736.061453] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7736.292829] Lustre: server umount lustre-MDT0001 complete [ 7747.688603] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 7750.281729] Key type lgssc unregistered [ 7750.512802] LNet: 238345:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7750.517761] LNetError: 238345:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7751.527323] LNet: Removed LNI 192.168.202.128@tcp [ 7752.752829] Key type .llcrypt unregistered [ 7752.754599] Key type ._llcrypt unregistered [ 7777.879260] Key type ._llcrypt registered [ 7777.880950] Key type .llcrypt registered [ 7778.013110] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 7779.356701] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7779.405580] alg: No test for adler32 (adler32-zlib) [ 7780.424109] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 7780.693289] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 7782.365287] Key type lgssc registered [ 7783.571367] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7792.821791] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7792.842457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7794.303811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7798.222486] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7804.906553] LustreError: 239814:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7804.920033] LustreError: 239814:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7806.554452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7806.702590] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7806.733959] Lustre: 239802:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7806.744976] Lustre: Permanently deactivating lustre-MDT0001 [ 7806.759307] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7807.007832] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7811.024212] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7813.117446] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7818.379680] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7824.589452] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7831.480582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7831.737283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7838.815181] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7839.922894] LustreError: 241655:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7839.941709] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7845.346668] LustreError: 241655:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7847.110201] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7850.488097] LustreError: 241653:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7851.581279] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7854.070168] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7854.295563] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7859.712830] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7859.734050] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7861.006421] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7868.352974] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7871.620797] Lustre: Permanently reactivating lustre-MDT0001 [ 7871.642785] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7871.652189] Lustre: Skipped 2 previous similar messages [ 7877.378743] Lustre: lustre-MDT0001: Client 01f87832-71ba-49cc-95c5-7be1c3030072 (at 192.168.202.28@tcp) reconnecting [ 7878.265645] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7878.291394] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7878.302369] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7878.334652] LustreError: 243386:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7878.336599] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7878.344945] LustreError: 243386:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7878.345095] LustreError: 243386:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.128@tcp: cfg command failed: rc = -108 [ 7878.345114] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7878.345114] [ 7878.348301] LustreError: 239807:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7882.477480] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7883.813612] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7889.161801] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7889.339837] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7893.255744] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7893.517476] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7896.007635] Lustre: Permanently deactivating lustre-MDT0001 [ 7896.028283] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7896.034671] Lustre: Skipped 2 previous similar messages [ 7905.957711] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7916.012326] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7916.020997] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7916.031251] Lustre: Skipped 2 previous similar messages [ 7916.047832] Lustre: Skipped 3 previous similar messages [ 7921.122170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7921.134165] Lustre: Skipped 3 previous similar messages [ 7921.379984] Lustre: server umount lustre-MDT0000 complete [ 7925.075141] LustreError: 243359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160062 with bad export cookie 14835331351749868225 [ 7925.085986] LustreError: 243359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7925.089698] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7925.489432] Lustre: server umount lustre-MDT0001 complete [ 7931.326732] Lustre: server umount lustre-OST0000 complete [ 7941.850869] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 03:07:58 (1788160078) [ 7951.974932] Lustre: server umount lustre-OST0001 complete [ 7958.266804] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 7967.596741] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 7999.893916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8000.131829] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8000.138215] Lustre: Skipped 2 previous similar messages [ 8000.153828] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8000.222362] Lustre: lustre-MDT0000: new disk, initializing [ 8000.295459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8000.305554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8004.655498] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 8014.168525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8014.245521] Lustre: 248114:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8014.316252] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8014.328352] Lustre: Skipped 1 previous similar message [ 8014.444798] Lustre: lustre-MDT0001: new disk, initializing [ 8014.595579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8014.679518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8014.692505] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8018.920461] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 8028.930565] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8035.107733] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8043.583547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8043.989640] Lustre: lustre-OST0000: new disk, initializing [ 8043.996815] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8044.018647] Lustre: 250086:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8044.100866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8050.795638] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 8052.850631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8052.870722] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8052.939623] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8059.868281] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8062.068819] LustreError: 248123:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8071.287220] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8071.441574] Lustre: lustre-OST0001: new disk, initializing [ 8071.445980] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8071.451776] Lustre: 251201:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8072.159131] LustreError: 248123:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8072.186736] LustreError: 248122:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8077.917757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8077.929849] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8078.047381] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8078.062596] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 8082.207569] LustreError: 248122:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8087.633496] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8089.636947] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8093.665144] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8093.665587] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8093.674228] LustreError: Skipped 1 previous similar message [ 8093.674884] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8093.690929] Lustre: Skipped 2 previous similar messages [ 8098.787142] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8098.794608] Lustre: Skipped 1 previous similar message [ 8103.913475] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8103.933677] Lustre: Skipped 2 previous similar messages [ 8105.951875] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8106.073089] Lustre: server umount lustre-OST0001 complete [ 8109.032665] LustreError: 250072:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8109.048235] LustreError: 250072:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8119.276039] LustreError: 250549:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8119.303377] LustreError: 250549:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8124.397682] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8124.423431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8124.425131] Lustre: Skipped 1 previous similar message [ 8124.443388] Lustre: Skipped 1 previous similar message [ 8129.613443] Lustre: server umount lustre-OST0000 complete [ 8134.625428] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8134.637835] Lustre: Skipped 1 previous similar message [ 8139.599478] Lustre: server umount lustre-MDT0000 complete [ 8139.752164] LustreError: 248128:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8139.772273] LustreError: 248128:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 8142.970510] LustreError: 248109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160280 with bad export cookie 14835331351749871032 [ 8142.973538] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8142.978540] LustreError: 248109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8149.440691] Lustre: server umount lustre-MDT0001 complete [ 8161.101911] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 8163.819868] Key type lgssc unregistered [ 8164.108909] LNet: 253236:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8164.112503] LNetError: 253236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8164.127455] LNet: Removed LNI 192.168.202.128@tcp [ 8164.819206] Key type .llcrypt unregistered [ 8164.821049] Key type ._llcrypt unregistered [ 8202.071800] Key type ._llcrypt registered [ 8202.074425] Key type .llcrypt registered [ 8202.239114] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 8203.806108] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8203.836750] alg: No test for adler32 (adler32-zlib) [ 8205.043421] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 8205.426615] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 8207.153439] Key type lgssc registered [ 8208.593202] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8216.877124] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8216.897526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8218.213340] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8218.236280] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8218.484258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8222.892696] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 8230.585095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8230.685047] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8230.713983] Lustre: 255443:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8230.918346] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8235.171297] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 8242.076398] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8247.840685] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8255.782051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8256.104172] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8256.319709] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8263.622928] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 8264.342544] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8271.335837] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8325.611559] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8325.618458] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8325.629453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8326.117870] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8326.131101] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8326.143501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8326.149208] Lustre: Skipped 1 previous similar message [ 8327.524317] Lustre: server umount lustre-OST0000 complete [ 8331.421851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8343.899233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8344.270203] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8346.353190] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8346.370521] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8349.368584] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 8351.528749] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8351.542487] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8355.580796] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8397.794547] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8397.807849] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8397.818881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8397.820925] Lustre: Skipped 1 previous similar message [ 8400.658600] Lustre: server umount lustre-OST0000 complete [ 8408.033705] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8408.039034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8408.047175] Lustre: Skipped 1 previous similar message [ 8408.062831] Lustre: Skipped 2 previous similar messages [ 8411.362985] Lustre: server umount lustre-MDT0000 complete [ 8415.027774] LustreError: 257306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160552 with bad export cookie 1492864553347431879 [ 8415.033757] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8415.039148] LustreError: 257306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8415.488603] Lustre: server umount lustre-MDT0001 complete [ 8425.499280] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 8428.352207] Key type lgssc unregistered [ 8428.755960] LNet: 260869:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8428.780463] LNetError: 260869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8428.805943] LNet: Removed LNI 192.168.202.128@tcp [ 8429.773676] Key type .llcrypt unregistered [ 8429.775542] Key type ._llcrypt unregistered [ 8450.577990] Key type ._llcrypt registered [ 8450.580436] Key type .llcrypt registered [ 8450.672759] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 8451.636398] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8451.660599] alg: No test for adler32 (adler32-zlib) [ 8452.678722] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 8452.927281] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 8454.599328] Key type lgssc registered [ 8455.895495] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8464.892637] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8464.928533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8466.517780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8470.487340] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 8476.645699] LustreError: 262266:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8476.675165] LustreError: 262266:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8479.920261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8480.545166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8485.995037] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 8494.916379] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8501.230595] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8508.874539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8509.341778] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8511.436947] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8515.878174] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 8523.762760] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8556.514594] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8556.522721] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8556.527532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8556.536651] Lustre: Skipped 1 previous similar message [ 8556.553752] Lustre: Skipped 1 previous similar message [ 8561.568511] Lustre: server umount lustre-OST0000 complete [ 8566.754032] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8566.756045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8566.769717] Lustre: Skipped 1 previous similar message [ 8571.878168] LustreError: 262266:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8571.899335] LustreError: 262266:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8571.982330] Lustre: server umount lustre-MDT0000 complete [ 8576.202541] LustreError: 262247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160714 with bad export cookie 9907866182944553681 [ 8576.221386] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8576.550210] Lustre: server umount lustre-MDT0001 complete [ 8587.426827] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 8590.839700] Key type lgssc unregistered [ 8591.225674] LNet: 266571:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8591.232376] LNetError: 266571:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8591.254947] LNet: Removed LNI 192.168.202.128@tcp [ 8592.168449] Key type .llcrypt unregistered [ 8592.179570] Key type ._llcrypt unregistered [ 8611.082084] Key type ._llcrypt registered [ 8611.084294] Key type .llcrypt registered [ 8611.335885] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 8613.508250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8613.567713] alg: No test for adler32 (adler32-zlib) [ 8614.714573] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 8615.040842] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 8616.776316] Key type lgssc registered [ 8618.601741] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8626.636321] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8626.664529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8628.075275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8632.015563] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 8637.411850] LustreError: 267796:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8637.422649] LustreError: 267796:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8640.136526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8640.458429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8644.418577] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 8650.957735] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8656.387671] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8663.207729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8663.517508] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8669.724694] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 8672.787196] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8677.372720] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8691.172201] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8691.181281] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8691.205459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8693.216468] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8693.222446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8693.224902] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8693.257614] Lustre: Skipped 1 previous similar message [ 8695.174588] Lustre: server umount lustre-OST0000 complete [ 8701.927220] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8701.938647] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8701.961473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8706.998229] Lustre: server umount lustre-MDT0000 complete [ 8708.576576] LustreError: 267792:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8708.602288] LustreError: 267792:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8712.638929] LustreError: 267777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160850 with bad export cookie 17718855490179692624 [ 8712.648963] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8712.663880] LustreError: 267777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8713.286485] Lustre: server umount lustre-MDT0001 complete [ 8727.594873] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 8730.460265] Key type lgssc unregistered [ 8730.721933] LNet: 271475:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8730.729372] LNetError: 271475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8730.759195] LNet: Removed LNI 192.168.202.128@tcp [ 8731.781210] Key type .llcrypt unregistered [ 8731.783460] Key type ._llcrypt unregistered [ 8757.876114] Key type ._llcrypt registered [ 8757.878458] Key type .llcrypt registered [ 8757.987889] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 8759.249068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8759.345105] alg: No test for adler32 (adler32-zlib) [ 8760.378472] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 8760.798970] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 8762.519168] Key type lgssc registered [ 8763.647240] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8773.297786] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8773.316051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8774.827908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8780.002241] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 8784.361020] LustreError: 272895:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8784.389487] LustreError: 272895:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8789.328356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8789.474882] LustreError: 272894:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8789.729725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8793.999555] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 8801.316606] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8807.850713] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8814.799459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8815.275276] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8819.400137] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8820.957592] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 8828.638370] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8855.521820] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8855.526207] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8855.531339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8855.541566] Lustre: Skipped 1 previous similar message [ 8855.546319] LustreError: Skipped 1 previous similar message [ 8860.651115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8860.665372] Lustre: Skipped 1 previous similar message [ 8861.233722] Lustre: server umount lustre-OST0000 complete [ 8865.760745] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8865.771839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8865.789667] Lustre: Skipped 1 previous similar message [ 8865.820441] Lustre: Skipped 2 previous similar messages [ 8870.880319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8870.890036] Lustre: Skipped 1 previous similar message [ 8871.481833] Lustre: server umount lustre-MDT0000 complete [ 8875.314508] LustreError: 274739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161013 with bad export cookie 12640958091608931745 [ 8875.325235] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8875.328091] LustreError: 274739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8875.623133] Lustre: server umount lustre-MDT0001 complete [ 8887.846905] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 8890.834793] Key type lgssc unregistered [ 8891.170154] LNet: 277199:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8891.181793] LNetError: 277199:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8891.202135] LNet: Removed LNI 192.168.202.128@tcp [ 8892.101396] Key type .llcrypt unregistered [ 8892.105663] Key type ._llcrypt unregistered [ 8910.852217] Key type ._llcrypt registered [ 8910.854568] Key type .llcrypt registered [ 8910.972391] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 8912.276099] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8912.323288] alg: No test for adler32 (adler32-zlib) [ 8913.325846] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 8913.598953] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 8915.319647] Key type lgssc registered [ 8916.553833] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8925.154332] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8925.195236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8926.733671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8931.691310] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 8936.935406] LustreError: 278436:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8936.967240] LustreError: 278436:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8937.961172] LustreError: 278438:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8940.062819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8940.655389] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8946.345966] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 8956.041898] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8963.007921] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8970.003047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8970.505648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8972.595657] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8976.970985] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 8983.939381] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9001.448993] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9001.454170] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9001.469195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9004.764612] Lustre: server umount lustre-OST0000 complete [ 9011.686780] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9011.708410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9011.715988] Lustre: Skipped 2 previous similar messages [ 9011.721631] Lustre: Skipped 1 previous similar message [ 9015.102938] Lustre: server umount lustre-MDT0000 complete [ 9016.809937] LustreError: 278440:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9016.841638] LustreError: 278440:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9019.478036] LustreError: 278403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161157 with bad export cookie 3936863811818274679 [ 9019.505589] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9020.032712] Lustre: server umount lustre-MDT0001 complete [ 9031.244941] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 9034.427669] Key type lgssc unregistered [ 9034.730519] LNet: 282139:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9034.734054] LNetError: 282139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9034.764414] LNet: Removed LNI 192.168.202.128@tcp [ 9035.895929] Key type .llcrypt unregistered [ 9035.898115] Key type ._llcrypt unregistered [ 9048.734619] Key type ._llcrypt registered [ 9048.741838] Key type .llcrypt registered [ 9048.864832] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9066.399095] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 9082.821372] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 9084.159093] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9084.258295] alg: No test for adler32 (adler32-zlib) [ 9085.299454] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 9085.536361] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 9087.175172] Key type lgssc registered [ 9089.071782] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9125.282761] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9125.337747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9126.728943] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9126.776453] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9126.894790] Lustre: lustre-MDT0000: new disk, initializing [ 9127.036060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9127.067668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9132.122854] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 9143.127624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9143.244713] Lustre: 285700:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9143.327173] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9143.335803] Lustre: Skipped 1 previous similar message [ 9143.509199] Lustre: lustre-MDT0001: new disk, initializing [ 9143.632056] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9143.675935] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9143.699832] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9148.829483] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 9157.623484] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9163.909957] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9169.960918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9170.202385] Lustre: lustre-OST0000: new disk, initializing [ 9170.209388] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9170.216923] Lustre: 287668:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9170.368069] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9176.806882] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 9179.182710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9179.194486] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9179.298874] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9187.093469] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9191.810877] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9192.068281] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9196.926956] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9197.216414] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9200.099336] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9200.102899] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9200.111982] LustreError: Skipped 1 previous similar message [ 9200.117173] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9200.122779] Lustre: Skipped 1 previous similar message [ 9200.135953] Lustre: Skipped 1 previous similar message [ 9205.232451] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9205.243821] Lustre: Skipped 1 previous similar message [ 9205.633195] Lustre: server umount lustre-OST0000 complete [ 9215.469976] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9215.479765] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9215.487317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9215.510387] Lustre: Skipped 1 previous similar message [ 9220.577160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9220.580941] Lustre: Skipped 1 previous similar message [ 9225.183135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9225.359537] Lustre: server umount lustre-MDT0000 complete [ 9225.696347] LustreError: 286762:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9225.709576] LustreError: 286762:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9229.128680] LustreError: 285692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161367 with bad export cookie 10843718228886304556 [ 9229.132756] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9229.137480] LustreError: 285692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9229.546714] Lustre: server umount lustre-MDT0001 complete [ 9239.474966] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 03:29:35 (1788161375) [ 9247.882684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9248.202077] LustreError: 290080:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9248.275980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9253.350448] LustreError: 290081:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9253.532528] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 9258.466801] LustreError: 290080:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9262.488117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9262.878496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9268.357454] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 9276.990134] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9283.307824] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9289.944951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9290.205139] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9296.990421] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 9305.346864] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9317.857844] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9317.866443] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9317.885393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9317.888866] Lustre: Skipped 2 previous similar messages [ 9317.915695] Lustre: Skipped 1 previous similar message [ 9321.688426] Lustre: server umount lustre-OST0000 complete [ 9328.100194] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9328.109299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9331.930593] Lustre: server umount lustre-MDT0000 complete [ 9333.216309] LustreError: 291050:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9333.239221] LustreError: 291050:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9336.824347] LustreError: 290062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161474 with bad export cookie 10843718228886305445 [ 9336.827361] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9336.843546] LustreError: 290062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9337.502449] Lustre: server umount lustre-MDT0001 complete [ 9349.803799] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 9352.817092] Key type lgssc unregistered [ 9353.223473] LNet: 293618:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9353.236468] LNetError: 293618:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9354.277701] LNet: Removed LNI 192.168.202.128@tcp [ 9355.297436] Key type .llcrypt unregistered [ 9355.299116] Key type ._llcrypt unregistered [ 9390.162678] Key type ._llcrypt registered [ 9390.165190] Key type .llcrypt registered [ 9390.268685] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 9405.587670] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 9406.760460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9406.797517] alg: No test for adler32 (adler32-zlib) [ 9408.013439] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 9408.398797] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 9410.151178] Key type lgssc registered [ 9411.194916] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9463.269765] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9463.339315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9464.522638] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9464.542618] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9464.633354] Lustre: lustre-MDT0000: new disk, initializing [ 9464.772453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9464.793456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9469.250037] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 9479.934741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9480.042690] Lustre: 297450:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9480.083534] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9480.086879] Lustre: Skipped 1 previous similar message [ 9480.184323] Lustre: lustre-MDT0001: new disk, initializing [ 9480.296478] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9480.355922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9480.368795] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9484.977858] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 9494.777148] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9500.773606] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9508.213711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9508.564390] Lustre: lustre-OST2710: new disk, initializing [ 9508.567079] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9508.573913] Lustre: 299417:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9508.668087] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9515.607992] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 9518.674252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9518.679538] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9518.863555] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9527.082624] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9534.757938] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9534.984210] Lustre: lustre-OST03e8: new disk, initializing [ 9534.987365] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9534.996327] Lustre: 300482:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9535.094205] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9542.080664] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 9543.805714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9543.816300] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9543.910436] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9550.050829] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9558.274299] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9558.464524] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9563.932382] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9564.158302] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9569.427915] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9569.659751] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9574.961798] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9575.186370] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9582.474356] Lustre: server umount lustre-MDT0000 complete [ 9582.564532] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9582.573205] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9585.127847] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9586.833382] LustreError: 298244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161724 with bad export cookie 8617101926381757532 [ 9586.864074] LustreError: 298244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9586.865775] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9587.386968] Lustre: server umount lustre-MDT0001 complete [ 9601.457163] Lustre: server umount lustre-OST2710 complete [ 9615.794420] Lustre: server umount lustre-OST03e8 complete [ 9635.214586] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 9647.610579] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 9688.168846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9688.470939] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9688.497423] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9688.591699] Lustre: lustre-MDT0000: new disk, initializing [ 9688.731318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9688.747362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9693.079058] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 9703.618695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9703.718462] Lustre: 305735:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9703.774686] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9703.779365] Lustre: Skipped 1 previous similar message [ 9703.883991] Lustre: lustre-MDT0001: new disk, initializing [ 9703.981142] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9704.014949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9704.025419] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9708.400475] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 9718.181504] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9725.479351] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9734.594695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9735.080937] Lustre: lustre-OST0000: new disk, initializing [ 9735.083843] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9735.088123] Lustre: 307704:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9735.188747] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9742.878361] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 9743.430783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9743.440729] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9743.543741] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9752.741818] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9759.264593] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9759.459383] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9765.888454] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9766.249980] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9769.440763] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9769.448980] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9769.454863] LustreError: Skipped 1 previous similar message [ 9769.457223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9769.487020] Lustre: Skipped 3 previous similar messages [ 9774.563099] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9774.582829] Lustre: Skipped 2 previous similar messages [ 9775.000715] Lustre: server umount lustre-OST0000 complete [ 9779.681102] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9779.702352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9779.704416] Lustre: Skipped 1 previous similar message [ 9779.730691] Lustre: Skipped 1 previous similar message [ 9784.807545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9786.028895] Lustre: server umount lustre-MDT0000 complete [ 9789.920959] LustreError: 306612:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9789.959305] LustreError: 306612:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 9790.505051] LustreError: 305726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161928 with bad export cookie 8617101926381759212 [ 9790.521118] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9790.931353] Lustre: server umount lustre-MDT0001 complete [ 9801.570236] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 03:38:57 (1788161937) [ 9803.468981] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9805.518925] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 03:39:01 (1788161941) [ 9809.432870] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 9839.592670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9839.974766] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9840.000769] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9840.122195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9845.133593] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 9855.900450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9856.061532] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9856.097512] Lustre: 311564:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9861.284139] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 9884.093732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9884.433892] Lustre: Denying initial registration attempt from nid 192.168.202.128@tcp, specified as failover [ 9884.443355] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9884.452137] LustreError: 313016:0:(tgt_mount.c:2688:server_fill_super()) Unable to start targets: -99 [ 9884.459471] LustreError: 313016:0:(tgt_mount.c:2206:server_put_super()) no obd lustre-OST0000 [ 9884.464822] LustreError: 313016:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9884.501772] Lustre: server umount lustre-OST0000 complete [ 9884.504425] LustreError: 313016:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9893.345467] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9893.381847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9893.388114] Lustre: Skipped 2 previous similar messages [ 9899.162477] Lustre: server umount lustre-MDT0000 complete [ 9902.848818] LustreError: 312313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788162040 with bad export cookie 8617101926381760094 [ 9902.854884] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9902.857548] LustreError: 312313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 9903.109637] Lustre: server umount lustre-MDT0001 complete [ 9917.429682] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 9921.959924] Key type lgssc unregistered [ 9922.273190] LNet: 314137:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9922.294055] LNetError: 314137:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9922.311597] LNet: Removed LNI 192.168.202.128@tcp [ 9923.351488] Key type .llcrypt unregistered [ 9923.353831] Key type ._llcrypt unregistered [ 9939.108474] Key type ._llcrypt registered [ 9939.110584] Key type .llcrypt registered [ 9939.198622] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 9940.475255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9940.607994] alg: No test for adler32 (adler32-zlib) [ 9941.627823] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 9941.827186] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 9943.472910] Key type lgssc registered [ 9944.701976] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9996.519266] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [10006.075775] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10006.094082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10007.412419] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10007.436892] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10007.614849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10013.255971] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [10022.551662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10022.685994] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10022.720647] Lustre: 316905:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10023.159807] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10028.893750] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [10041.130105] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10050.355413] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10072.181864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10072.553426] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10072.801408] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10080.163776] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [10089.027819] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10092.001674] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10092.006863] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10092.033551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10092.525932] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10092.526508] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10092.563324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10092.574597] Lustre: Skipped 1 previous similar message [10097.639990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10097.651140] Lustre: Skipped 1 previous similar message [10102.765027] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10102.783225] Lustre: Skipped 1 previous similar message [10105.311192] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10105.434588] Lustre: server umount lustre-OST0000 complete [10110.450821] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10110.460387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10110.469560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10112.993482] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10115.622039] Lustre: server umount lustre-MDT0000 complete [10119.247727] LustreError: 318775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788162257 with bad export cookie 1305057211404894628 [10119.254937] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10119.258482] LustreError: 318775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10119.578708] Lustre: server umount lustre-MDT0001 complete [10131.325756] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [10133.590946] Key type lgssc unregistered [10133.887889] LNet: 320484:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10133.895331] LNetError: 320484:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10133.914635] LNet: Removed LNI 192.168.202.128@tcp [10134.874738] Key type .llcrypt unregistered [10134.878688] Key type ._llcrypt unregistered [10162.567310] Key type ._llcrypt registered [10162.569679] Key type .llcrypt registered [10162.723182] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [10163.034531] Key type .llcrypt unregistered [10163.037623] Key type ._llcrypt unregistered [10187.407720] Key type ._llcrypt registered [10187.421927] Key type .llcrypt registered [10187.621198] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [10188.893651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10189.301321] alg: No test for adler32 (adler32-zlib) [10190.732883] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [10191.077837] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [10192.783193] Key type lgssc registered [10194.230136] Lustre: Echo OBD driver; http://www.lustre.org/ [10229.676620] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing lnet_if_list [10231.137810] LNet: Added LNI 192.168.202.128@tcp1 [8/256/0/180] [10245.954835] loop0: detected capacity change from 0 to 204800000 [10246.194578] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10246.211890] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10246.233675] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10265.149604] loop0: detected capacity change from 0 to 204800000 [10265.407974] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10265.424681] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10265.439814] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10284.375972] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10284.398753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10285.799973] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10285.817447] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10285.985187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10290.805971] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [10300.240744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10300.435802] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10300.502346] Lustre: 324410:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10300.794123] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10305.527283] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [10314.031340] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10319.158220] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10323.933562] loop0: detected capacity change from 0 to 204800000 [10342.836761] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10343.512025] Lustre: sv23120-OST0000: new disk, initializing [10343.522515] Lustre: srv-sv23120-OST0000: No data found on store. Initialize space. [10343.543754] Lustre: 326492:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10343.699308] Lustre: sv23120-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10352.544106] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [10364.814748] loop1: detected capacity change from 0 to 204800000 [10382.989712] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10383.169231] Lustre: sv23120-OST0001: new disk, initializing [10383.172590] Lustre: srv-sv23120-OST0001: No data found on store. Initialize space. [10383.179076] Lustre: 327608:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10383.261661] Lustre: sv23120-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10390.214606] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [10404.923702] Lustre: server umount sv23120-OST0001 complete [10414.619724] Lustre: server umount sv23120-OST0000 complete [10432.481533] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10432.487509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10432.502039] Lustre: Skipped 1 previous similar message [10432.517694] Lustre: Skipped 1 previous similar message [10435.638621] Lustre: server umount lustre-MDT0000 complete [10439.516690] LustreError: 324402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788162577 with bad export cookie 7564062097387661068 [10439.522472] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10439.531059] LustreError: 324402:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10440.111985] Lustre: server umount lustre-MDT0001 complete [10455.051834] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [10459.748544] Key type lgssc unregistered [10460.160074] LNet: 330363:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10460.173133] LNetError: 330363:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10460.184620] LNet: Removed LNI 192.168.202.128@tcp [10461.235203] LNet: Removed LNI 192.168.202.128@tcp1 [10462.424272] Key type .llcrypt unregistered [10462.426858] Key type ._llcrypt unregistered [10485.579477] Key type ._llcrypt registered [10485.594308] Key type .llcrypt registered [10485.786719] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [10486.094481] Key type .llcrypt unregistered [10486.096815] Key type ._llcrypt unregistered [10503.007343] Key type ._llcrypt registered [10503.009421] Key type .llcrypt registered [10503.189419] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [10504.623511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10504.694901] alg: No test for adler32 (adler32-zlib) [10505.874616] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [10506.075056] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [10507.752937] Key type lgssc registered [10509.048764] Lustre: Echo OBD driver; http://www.lustre.org/ [10533.223806] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing lnet_if_list [10534.373672] LNet: Added LNI 192.168.202.128@tcp1 [8/256/0/180] [10546.391736] loop0: detected capacity change from 0 to 204800000 [10546.642707] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10546.660474] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10546.682379] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10565.597269] loop0: detected capacity change from 0 to 204800000 [10565.833835] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10565.842846] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10565.853269] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10587.250786] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10587.291902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10588.823566] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10588.901954] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10589.232761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10594.931112] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [10605.771282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10605.887908] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10605.916827] Lustre: 334186:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10606.197210] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10613.176959] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [10622.711398] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10630.092214] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10636.875508] loop0: detected capacity change from 0 to 204800000 [10658.028798] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10658.300677] Lustre: sv8621-OST0000: new disk, initializing [10658.305590] Lustre: srv-sv8621-OST0000: No data found on store. Initialize space. [10658.314261] Lustre: 336269:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10658.381109] Lustre: sv8621-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10665.973312] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [10678.170352] loop1: detected capacity change from 0 to 204800000 [10698.675718] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10698.871291] Lustre: sv8621-OST0001: new disk, initializing [10698.892215] Lustre: srv-sv8621-OST0001: No data found on store. Initialize space. [10698.905492] Lustre: 337385:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10699.071392] Lustre: sv8621-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10708.455736] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [10725.871787] Lustre: server umount sv8621-OST0001 complete [10737.648485] Lustre: server umount sv8621-OST0000 complete [10752.483229] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10752.498409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10752.498993] Lustre: Skipped 1 previous similar message [10752.511893] Lustre: Skipped 1 previous similar message [10756.765943] Lustre: server umount lustre-MDT0000 complete [10760.112523] LustreError: 334178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788162897 with bad export cookie 9491246183844736121 [10760.113976] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10760.129046] LustreError: 334178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10760.468261] Lustre: server umount lustre-MDT0001 complete [10774.530491] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [10778.033276] Key type lgssc unregistered [10778.434257] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.28@tcp [10778.467941] LNet: 340141:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10778.477063] LNetError: 340141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10779.487310] LNet: Removed LNI 192.168.202.128@tcp [10779.489335] LNetError: 340141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10779.505675] LNet: Removed LNI 192.168.202.128@tcp1 [10780.531339] Key type .llcrypt unregistered [10780.533711] Key type ._llcrypt unregistered [10802.662436] Key type ._llcrypt registered [10802.666673] Key type .llcrypt registered [10802.824385] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [10803.181410] Key type .llcrypt unregistered [10803.186365] Key type ._llcrypt unregistered [10820.875702] Key type ._llcrypt registered [10820.881371] Key type .llcrypt registered [10821.008317] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [10822.136662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10822.178641] alg: No test for adler32 (adler32-zlib) [10823.609913] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [10824.027840] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [10825.825070] Key type lgssc registered [10827.281685] Lustre: Echo OBD driver; http://www.lustre.org/ [10851.977875] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing lnet_if_list [10853.069868] LNet: Added LNI 192.168.202.128@tcp1 [8/256/0/180] [10863.925495] loop0: detected capacity change from 0 to 204800000 [10864.026707] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10864.034807] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10864.043264] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10883.932302] loop0: detected capacity change from 0 to 204800000 [10884.456157] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10884.493360] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10884.554449] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10905.876436] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10905.891969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10907.142898] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10907.167958] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10907.324092] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10913.251514] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [10924.399339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10924.600205] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10924.658478] Lustre: 343962:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10925.069564] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10931.086432] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [10939.984356] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10947.136119] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10953.478646] loop0: detected capacity change from 0 to 204800000 [10973.750033] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10974.012879] Lustre: sv26581-OST0000: new disk, initializing [10974.023589] Lustre: srv-sv26581-OST0000: No data found on store. Initialize space. [10974.028184] Lustre: 346043:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10974.165616] Lustre: sv26581-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10982.483514] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [11003.513425] Lustre: server umount sv26581-OST0000 complete [11021.803354] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11021.811603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11021.822383] Lustre: Skipped 1 previous similar message [11021.845126] Lustre: Skipped 1 previous similar message [11026.111576] Lustre: server umount lustre-MDT0000 complete [11030.970794] LustreError: 343953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788163168 with bad export cookie 11264259206713175291 [11030.976238] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11030.991631] LustreError: 343953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11031.425813] Lustre: server umount lustre-MDT0001 complete [11045.779907] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [11048.802508] Key type lgssc unregistered [11049.215504] LNet: 348497:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11049.225268] LNetError: 348497:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11049.232528] LNet: Removed LNI 192.168.202.128@tcp [11050.278802] LNet: Removed LNI 192.168.202.128@tcp1 [11051.459620] Key type .llcrypt unregistered [11051.464807] Key type ._llcrypt unregistered [11073.098837] Key type ._llcrypt registered [11073.107772] Key type .llcrypt registered [11073.340692] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [11073.759821] Key type .llcrypt unregistered [11073.761863] Key type ._llcrypt unregistered [11093.975438] Key type ._llcrypt registered [11093.977669] Key type .llcrypt registered [11094.139144] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [11094.969525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11095.025049] alg: No test for adler32 (adler32-zlib) [11096.681933] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [11097.004907] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [11098.719276] Key type lgssc registered [11099.934421] Lustre: Echo OBD driver; http://www.lustre.org/ [11131.154437] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing lnet_if_list [11133.318702] LNet: Added LNI 192.168.202.128@tcp1 [8/256/0/180] [11148.041167] loop0: detected capacity change from 0 to 204800000 [11148.495109] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11148.513479] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11148.558737] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11168.206211] loop0: detected capacity change from 0 to 204800000 [11168.518578] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11168.527665] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11168.543371] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11188.741539] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11188.772483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11190.239561] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11190.298353] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11190.610365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11196.800910] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [11207.643434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11207.761952] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11207.816713] Lustre: 352380:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11208.077022] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11212.963879] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [11221.886658] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11229.003384] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11236.302174] loop0: detected capacity change from 0 to 204800000 [11257.158377] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11257.671513] Lustre: sv27299-OST0000: new disk, initializing [11257.676844] Lustre: srv-sv27299-OST0000: No data found on store. Initialize space. [11257.691667] Lustre: 354462:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11257.797536] Lustre: sv27299-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11265.961799] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [11287.262312] Lustre: server umount sv27299-OST0000 complete [11305.440274] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11305.445426] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11305.478305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11305.499971] Lustre: Skipped 1 previous similar message [11305.516503] Lustre: Skipped 1 previous similar message [11310.566419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11310.569923] Lustre: Skipped 1 previous similar message [11315.683070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11315.697336] Lustre: Skipped 1 previous similar message [11318.240856] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11318.619813] Lustre: server umount lustre-MDT0000 complete [11320.800931] LustreError: 354436:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11320.823738] LustreError: 354436:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11322.937715] LustreError: 352375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788163460 with bad export cookie 12296867690717092282 [11322.942953] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11322.953961] LustreError: 352375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11323.706370] Lustre: server umount lustre-MDT0001 complete [11341.465545] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [11345.082772] Key type lgssc unregistered [11345.520994] LNet: 356920:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11345.528115] LNetError: 356920:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11345.533850] LNet: Removed LNI 192.168.202.128@tcp [11346.597629] LNet: Removed LNI 192.168.202.128@tcp1 [11348.127551] Key type .llcrypt unregistered [11348.131528] Key type ._llcrypt unregistered [11373.869343] Key type ._llcrypt registered [11373.871793] Key type .llcrypt registered [11374.002082] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [11374.371464] Key type .llcrypt unregistered [11374.374667] Key type ._llcrypt unregistered [11393.374212] Key type ._llcrypt registered [11393.376987] Key type .llcrypt registered [11393.552684] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [11394.771361] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11394.830539] alg: No test for adler32 (adler32-zlib) [11396.107116] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [11396.415667] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [11398.095173] Key type lgssc registered [11399.831997] Lustre: Echo OBD driver; http://www.lustre.org/ [11427.625758] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing lnet_if_list [11429.437362] LNet: Added LNI 192.168.202.128@tcp1 [8/256/0/180] [11443.546093] loop0: detected capacity change from 0 to 204800000 [11443.897918] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11443.923672] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11443.954684] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11463.180506] loop0: detected capacity change from 0 to 204800000 [11463.444584] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11463.457955] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11463.481059] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11482.858477] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11482.867235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11484.170769] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11484.201285] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11484.412411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11489.584702] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [11499.878650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11499.953810] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11499.983864] Lustre: 360745:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11500.190270] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11505.568207] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [11514.481104] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11520.216995] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11526.092435] loop0: detected capacity change from 0 to 204800000 [11544.293015] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11544.445729] Lustre: sv6772-OST0000: new disk, initializing [11544.447988] Lustre: srv-sv6772-OST0000: No data found on store. Initialize space. [11544.451162] Lustre: 362822:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11544.498624] Lustre: sv6772-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11552.311358] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [11565.696257] loop1: detected capacity change from 0 to 204800000 [11585.510306] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11585.808480] Lustre: sv6772-OST0001: new disk, initializing [11585.819429] Lustre: srv-sv6772-OST0001: No data found on store. Initialize space. [11585.826401] Lustre: 363993:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11585.899159] Lustre: sv6772-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11593.231165] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [11610.000947] Lustre: server umount sv6772-OST0001 complete [11622.301996] Lustre: server umount sv6772-OST0000 complete [11638.752643] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11638.758521] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11638.783719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11639.776176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11639.779847] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11644.349059] Lustre: server umount lustre-MDT0000 complete [11648.526705] LustreError: 360738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788163786 with bad export cookie 9779549985938284300 [11648.534633] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11648.545450] LustreError: 360738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11649.030985] Lustre: server umount lustre-MDT0001 complete [11664.288358] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [11666.978773] Key type lgssc unregistered [11667.322959] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.28@tcp [11667.373174] LNet: 366750:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11667.377509] LNetError: 366750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11667.386410] LNet: Removed LNI 192.168.202.128@tcp [11668.455477] LNet: Removed LNI 192.168.202.128@tcp1 [11669.589245] Key type .llcrypt unregistered [11669.591314] Key type ._llcrypt unregistered [11689.794386] Key type ._llcrypt registered [11689.795624] Key type .llcrypt registered [11689.902721] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [11690.259451] Key type .llcrypt unregistered [11690.261342] Key type ._llcrypt unregistered [11709.371531] Key type ._llcrypt registered [11709.382330] Key type .llcrypt registered [11709.562512] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [11711.056166] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11711.288521] alg: No test for adler32 (adler32-zlib) [11712.370486] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [11712.621237] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [11714.343745] Key type lgssc registered [11715.673907] Lustre: Echo OBD driver; http://www.lustre.org/ [11738.438082] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing lnet_if_list [11739.712800] LNet: Added LNI 192.168.202.128@tcp1 [8/256/0/180] [11776.455928] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [11780.839818] Key type lgssc unregistered [11781.253173] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.28@tcp [11781.309976] LNet: 370734:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11781.315416] LNetError: 370734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11781.322621] LNet: Removed LNI 192.168.202.128@tcp [11781.355259] LNet: Removed LNI 192.168.202.128@tcp1 [11782.668894] Key type .llcrypt unregistered [11782.676877] Key type ._llcrypt unregistered [11805.066791] Key type ._llcrypt registered [11805.068508] Key type .llcrypt registered [11805.164381] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [11805.376488] Key type .llcrypt unregistered [11805.378157] Key type ._llcrypt unregistered [11823.416433] Key type ._llcrypt registered [11823.420276] Key type .llcrypt registered [11823.558489] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [11825.353677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11825.379520] alg: No test for adler32 (adler32-zlib) [11826.543846] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [11826.870859] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [11828.567179] Key type lgssc registered [11829.971603] Lustre: Echo OBD driver; http://www.lustre.org/ [11855.099950] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing lnet_if_list [11856.462681] LNet: Added LNI 192.168.202.128@tcp1 [8/256/0/180] [11858.484771] LNet: Added LNI 192.168.202.128@tcp2 [8/256/0/180] [11870.254168] loop0: detected capacity change from 0 to 204800000 [11870.420186] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11870.433275] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11870.440417] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11889.001849] loop0: detected capacity change from 0 to 204800000 [11889.260263] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11889.278332] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11889.301814] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11909.372565] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11909.400366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11910.634443] LNetError: 372063:0:(peer.c:2151:lnet_peer_add_nid()) Peer 192.168.202.128@tcp NID 192.168.202.128@tcp is already locked with peer 192.168.202.128@tcp [11910.835476] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11910.880382] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11911.107837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11917.033142] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [11927.215290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11927.316916] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11927.362413] Lustre: 374654:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11927.586406] LNetError: 372063:0:(peer.c:2151:lnet_peer_add_nid()) Peer 192.168.202.128@tcp NID 192.168.202.128@tcp is already locked with peer 192.168.202.128@tcp [11927.592039] LNetError: 372063:0:(peer.c:2151:lnet_peer_add_nid()) Skipped 1 previous similar message [11927.664110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11931.367403] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [11934.797941] LNetError: 372063:0:(peer.c:2151:lnet_peer_add_nid()) Peer 192.168.202.128@tcp NID 192.168.202.128@tcp is already locked with peer 192.168.202.128@tcp [11934.825941] LNetError: 372063:0:(peer.c:2151:lnet_peer_add_nid()) Skipped 2 previous similar messages [11941.328795] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11948.355589] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11954.633029] loop0: detected capacity change from 0 to 204800000 [11974.773485] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11975.218422] Lustre: sv8868-OST0000: new disk, initializing [11975.226798] Lustre: srv-sv8868-OST0000: No data found on store. Initialize space. [11975.231719] Lustre: 376731:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11975.299862] Lustre: sv8868-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11982.367116] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [11994.795057] loop1: detected capacity change from 0 to 204800000 [12014.191870] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12014.367276] Lustre: sv8868-OST0001: new disk, initializing [12014.374457] Lustre: srv-sv8868-OST0001: No data found on store. Initialize space. [12014.383308] Lustre: 377900:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12014.455859] Lustre: sv8868-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12021.556139] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [12036.866379] Lustre: server umount sv8868-OST0001 complete [12052.633326] Lustre: server umount sv8868-OST0000 complete [12070.885120] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12070.898419] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12070.914193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12072.933410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12072.933879] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12075.092846] Lustre: server umount lustre-MDT0000 complete [12080.058745] LustreError: 374645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788164217 with bad export cookie 9596148456124667881 [12080.069507] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12080.075771] LustreError: 374645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12080.677728] Lustre: server umount lustre-MDT0001 complete [12096.546169] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [12100.367426] Key type lgssc unregistered [12100.862845] LNet: 380659:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12100.878646] LNetError: 380659:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12101.920698] LNet: Removed LNI 192.168.202.128@tcp [12101.931968] LNetError: 380659:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12101.952815] LNet: Removed LNI 192.168.202.128@tcp1 [12103.015295] LNet: Removed LNI 192.168.202.128@tcp2 [12104.248855] Key type .llcrypt unregistered [12104.250577] Key type ._llcrypt unregistered [12128.104426] Key type ._llcrypt registered [12128.106745] Key type .llcrypt registered [12128.284975] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [12128.754299] Key type .llcrypt unregistered [12128.757803] Key type ._llcrypt unregistered [12149.651482] Key type ._llcrypt registered [12149.654044] Key type .llcrypt registered [12149.876949] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [12151.508391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12151.652162] alg: No test for adler32 (adler32-zlib) [12152.777230] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [12153.133655] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [12154.855226] Key type lgssc registered [12156.434230] Lustre: Echo OBD driver; http://www.lustre.org/ [12181.702425] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing lnet_if_list [12183.405828] LNet: Added LNI 192.168.202.128@tcp1 [8/256/0/180] [12195.293601] loop0: detected capacity change from 0 to 204800000 [12195.582983] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12195.593407] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12195.613705] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12213.682893] loop0: detected capacity change from 0 to 204800000 [12213.888135] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12213.910222] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12213.929350] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12233.265975] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12233.282076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12234.624925] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12234.656853] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12234.838529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12239.726873] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [12248.327271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12248.426355] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12248.493036] Lustre: 384536:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12248.586207] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [12248.737763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12252.862855] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [12261.727460] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12268.274981] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12277.041601] loop0: detected capacity change from 0 to 204800000 [12298.607533] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12298.791494] Lustre: sv20167-OST0000: new disk, initializing [12298.795354] Lustre: srv-sv20167-OST0000: No data found on store. Initialize space. [12298.804775] Lustre: 386613:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12298.864898] Lustre: sv20167-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12306.351235] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [12317.003786] loop1: detected capacity change from 0 to 204800000 [12336.399867] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12336.837641] Lustre: sv20167-OST0001: new disk, initializing [12336.840635] Lustre: srv-sv20167-OST0001: No data found on store. Initialize space. [12336.846966] Lustre: 387732:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12336.890775] Lustre: sv20167-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12343.918951] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [12360.195564] Lustre: server umount sv20167-OST0001 complete [12373.042150] Lustre: server umount sv20167-OST0000 complete [12387.297055] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12387.300497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12387.308526] Lustre: Skipped 1 previous similar message [12387.328683] Lustre: Skipped 1 previous similar message [12392.113249] Lustre: server umount lustre-MDT0000 complete [12396.608477] LustreError: 385292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788164534 with bad export cookie 8176845772226966909 [12396.614357] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12397.102755] Lustre: server umount lustre-MDT0001 complete [12413.185409] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [12416.654780] Key type lgssc unregistered [12416.786353] LNet: 1 peer NIs in recovery (showing 1): 192.168.202.28@tcp [12416.807328] LNet: 390488:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12416.811225] LNetError: 390488:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12416.814590] LNet: Removed LNI 192.168.202.128@tcp [12417.843924] LNet: Removed LNI 192.168.202.128@tcp1 [12418.554691] Key type .llcrypt unregistered [12418.556219] Key type ._llcrypt unregistered [12440.915997] Key type ._llcrypt registered [12440.923667] Key type .llcrypt registered [12441.087600] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [12441.483355] Key type .llcrypt unregistered [12441.485675] Key type ._llcrypt unregistered [12451.366623] Key type ._llcrypt registered [12451.371954] Key type .llcrypt registered [12451.538488] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [12451.989310] Key type .llcrypt unregistered [12451.994164] Key type ._llcrypt unregistered [12467.736423] Key type ._llcrypt registered [12467.738686] Key type .llcrypt registered [12467.846866] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [12468.021316] Key type .llcrypt unregistered [12468.023213] Key type ._llcrypt unregistered [12490.078537] Key type ._llcrypt registered [12490.084352] Key type .llcrypt registered [12490.217369] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [12490.518378] Key type .llcrypt unregistered [12490.520244] Key type ._llcrypt unregistered [12510.044333] Key type ._llcrypt registered [12510.049327] Key type .llcrypt registered [12510.188585] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [12512.046354] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12512.254766] alg: No test for adler32 (adler32-zlib) [12513.625434] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [12513.904121] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [12515.671313] Key type lgssc registered [12517.065796] Lustre: Echo OBD driver; http://www.lustre.org/ [12542.609115] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing lnet_if_list [12544.056170] LNet: Added LNI 192.168.202.128@tcp1 [8/256/0/180] [12556.257301] loop0: detected capacity change from 0 to 204800000 [12556.511528] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12556.531204] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12556.545932] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12575.505852] loop0: detected capacity change from 0 to 204800000 [12575.880667] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12575.913046] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12575.938524] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12595.307074] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12595.350868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12596.638719] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12596.652711] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12596.756792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12601.337925] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [12611.482034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12611.673276] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12611.722865] Lustre: 395947:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12611.963771] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12617.460173] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [12626.271499] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12633.129967] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12639.286419] loop0: detected capacity change from 0 to 204800000 [12659.375893] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12659.656509] Lustre: sv16849-OST0000: new disk, initializing [12659.659078] Lustre: srv-sv16849-OST0000: No data found on store. Initialize space. [12659.665536] Lustre: 398028:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12659.704191] Lustre: sv16849-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12666.684816] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [12677.551784] loop1: detected capacity change from 0 to 204800000 [12697.136356] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12697.402498] Lustre: sv16849-OST0001: new disk, initializing [12697.408952] Lustre: srv-sv16849-OST0001: No data found on store. Initialize space. [12697.421633] Lustre: 399146:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12697.474853] Lustre: sv16849-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12704.603581] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [12723.218796] Lustre: 400137:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [12723.223751] Lustre: 400137:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [12723.226757] Lustre: 400137:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [12723.232058] Lustre: 400137:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [12723.237538] Lustre: 400137:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [12723.242353] Lustre: 400137:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [12725.415844] Lustre: server umount sv16849-OST0001 complete [12735.874722] Lustre: server umount sv16849-OST0000 complete [12748.773274] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12748.785883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12748.790428] Lustre: Skipped 1 previous similar message [12753.900072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12755.065920] Lustre: server umount lustre-MDT0000 complete [12759.114367] LustreError: 395942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788164897 with bad export cookie 3169083632275980673 [12759.119192] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12759.124593] LustreError: 395942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12759.600472] Lustre: server umount lustre-MDT0001 complete [12772.776428] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [12775.509345] Key type lgssc unregistered [12775.807172] LNet: 402143:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12775.819177] LNetError: 402143:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12775.826594] LNet: Removed LNI 192.168.202.128@tcp [12776.870300] LNet: Removed LNI 192.168.202.128@tcp1 [12777.675606] Key type .llcrypt unregistered [12777.677658] Key type ._llcrypt unregistered [12798.011161] Key type ._llcrypt registered [12798.013400] Key type .llcrypt registered [12798.119360] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [12798.404404] Key type .llcrypt unregistered [12798.407745] Key type ._llcrypt unregistered [12816.047977] Key type ._llcrypt registered [12816.050281] Key type .llcrypt registered [12816.131304] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [12817.318583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12817.468543] alg: No test for adler32 (adler32-zlib) [12818.453845] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [12818.706475] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [12820.431684] Key type lgssc registered [12821.939910] Lustre: Echo OBD driver; http://www.lustre.org/ [12845.605848] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing lnet_if_list [12846.631503] LNet: Added LNI 192.168.202.128@tcp1 [8/256/0/180] [12856.781638] loop0: detected capacity change from 0 to 204800000 [12857.022038] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12857.037370] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12857.055690] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12875.206837] loop0: detected capacity change from 0 to 204800000 [12875.375876] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12875.388245] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12875.406568] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12894.168292] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12894.201832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12895.552129] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12895.581955] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12895.807650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12900.619670] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [12909.797406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12909.898541] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12909.935845] Lustre: 405968:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12910.205912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12915.552806] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [12924.468569] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12930.897385] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12937.117892] loop0: detected capacity change from 0 to 204800000 [12956.333600] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12956.650776] Lustre: sv28288-OST0000: new disk, initializing [12956.657377] Lustre: srv-sv28288-OST0000: No data found on store. Initialize space. [12956.675428] Lustre: 408044:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12956.745699] Lustre: sv28288-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12963.173365] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [12972.660209] loop1: detected capacity change from 0 to 204800000 [12991.167951] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12991.449671] Lustre: sv28288-OST0001: new disk, initializing [12991.459449] Lustre: srv-sv28288-OST0001: No data found on store. Initialize space. [12991.471405] Lustre: 409161:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12991.593920] Lustre: sv28288-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12997.178762] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [13013.732934] Lustre: 410152:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [13013.764345] Lustre: 410152:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [13013.789328] Lustre: 410152:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [13013.800226] Lustre: 410152:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [13013.819011] Lustre: 410152:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [13013.825427] Lustre: 410152:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [13016.335351] Lustre: server umount sv28288-OST0001 complete [13030.863816] Lustre: server umount sv28288-OST0000 complete [13045.730212] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13045.752432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13045.753099] Lustre: Skipped 1 previous similar message [13045.773070] Lustre: Skipped 1 previous similar message [13049.947211] Lustre: server umount lustre-MDT0000 complete [13053.685994] LustreError: 405958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788165191 with bad export cookie 12804408988259026959 [13053.691499] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13053.695678] LustreError: 405958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13053.984851] Lustre: server umount lustre-MDT0001 complete [13067.473530] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [13072.685959] Key type lgssc unregistered [13073.100538] LNet: 412162:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13073.125283] LNetError: 412162:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13073.143988] LNet: Removed LNI 192.168.202.128@tcp [13074.247923] LNet: Removed LNI 192.168.202.128@tcp1 [13075.988554] Key type .llcrypt unregistered [13075.990074] Key type ._llcrypt unregistered [13101.596467] Key type ._llcrypt registered [13101.598408] Key type .llcrypt registered [13101.654373] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [13101.877416] Key type .llcrypt unregistered [13101.879179] Key type ._llcrypt unregistered [13118.738518] Key type ._llcrypt registered [13118.740931] Key type .llcrypt registered [13118.829459] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [13119.751877] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13119.871811] alg: No test for adler32 (adler32-zlib) [13121.112767] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [13121.579448] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [13123.319141] Key type lgssc registered [13124.463882] Lustre: Echo OBD driver; http://www.lustre.org/ [13147.423720] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing lnet_if_list [13148.990965] LNet: Added LNI 192.168.202.128@tcp1 [8/256/0/180] [13161.806638] loop0: detected capacity change from 0 to 204800000 [13162.054149] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13162.073128] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13162.096672] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13185.524772] loop0: detected capacity change from 0 to 204800000 [13185.684754] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13185.794872] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13185.811423] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13204.939524] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13204.989121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13206.401926] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [13206.435749] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13206.688119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13211.356148] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [13222.473373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13222.613473] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13222.704894] Lustre: 415989:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13223.184740] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13228.197986] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [13236.545761] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13243.282921] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13248.510786] loop0: detected capacity change from 0 to 204800000 [13267.759103] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13268.168557] Lustre: sv15535-OST0000: new disk, initializing [13268.173569] Lustre: srv-sv15535-OST0000: No data found on store. Initialize space. [13268.189772] Lustre: 418069:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13268.285771] Lustre: sv15535-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13275.491149] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [13285.986797] loop1: detected capacity change from 0 to 204800000 [13305.379293] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13305.560348] Lustre: sv15535-OST0001: new disk, initializing [13305.569668] Lustre: srv-sv15535-OST0001: No data found on store. Initialize space. [13305.588258] Lustre: 419186:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13305.671921] Lustre: sv15535-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13313.122191] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [13331.582973] Lustre: 420275:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [13331.591447] Lustre: 420275:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [13331.597896] Lustre: 420275:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [13331.608291] Lustre: 420275:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [13331.617692] Lustre: 420275:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [13331.623938] Lustre: 420275:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [13332.820929] Lustre: 420323:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [13332.833525] Lustre: 420323:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 255 previous similar messages [13332.841590] Lustre: 420323:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [13332.846410] Lustre: 420323:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 255 previous similar messages [13332.854610] Lustre: 420323:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [13332.860236] Lustre: 420323:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 255 previous similar messages [13332.868493] Lustre: 420323:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [13332.876349] Lustre: 420323:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 255 previous similar messages [13332.881074] Lustre: 420323:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [13332.886375] Lustre: 420323:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 255 previous similar messages [13332.894038] Lustre: 420323:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [13332.898151] Lustre: 420323:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 255 previous similar messages [13336.125789] Lustre: server umount sv15535-OST0001 complete [13346.967738] Lustre: server umount sv15535-OST0000 complete [13361.633700] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13361.645569] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13361.667804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13363.175766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13363.177675] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13366.674896] Lustre: server umount lustre-MDT0000 complete [13371.320357] LustreError: 415981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788165509 with bad export cookie 13492189910854392067 [13371.322302] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13371.338732] LustreError: 415981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13371.763086] Lustre: server umount lustre-MDT0001 complete [13384.518565] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [13387.646977] Key type lgssc unregistered [13388.038078] LNet: 422331:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13388.045101] LNetError: 422331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13388.049988] LNet: Removed LNI 192.168.202.128@tcp [13389.102677] LNet: Removed LNI 192.168.202.128@tcp1 [13390.267253] Key type .llcrypt unregistered [13390.270716] Key type ._llcrypt unregistered [13411.842311] Key type ._llcrypt registered [13411.844145] Key type .llcrypt registered [13411.939140] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [13412.166397] Key type .llcrypt unregistered [13412.168685] Key type ._llcrypt unregistered [13430.878683] Key type ._llcrypt registered [13430.883594] Key type .llcrypt registered [13430.991358] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [13431.991741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13432.141049] alg: No test for adler32 (adler32-zlib) [13433.328880] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [13433.644238] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [13435.367491] Key type lgssc registered [13436.570038] Lustre: Echo OBD driver; http://www.lustre.org/ [13458.989766] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing lnet_if_list [13460.331733] LNet: Added LNI 192.168.202.128@tcp1 [8/256/0/180] [13471.278219] loop0: detected capacity change from 0 to 204800000 [13471.527161] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13471.539033] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13471.555751] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13489.697606] loop0: detected capacity change from 0 to 204800000 [13490.003450] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13490.029835] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13490.050042] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13508.614986] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13508.629695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13510.057435] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [13510.130338] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13510.375781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13514.554018] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [13521.429856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13521.521892] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13521.553917] Lustre: 426160:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13521.793768] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13526.581394] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [13535.008482] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13541.785713] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13548.667645] loop0: detected capacity change from 0 to 204800000 [13568.055878] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13568.326946] Lustre: sv2857-OST0000: new disk, initializing [13568.330414] Lustre: srv-sv2857-OST0000: No data found on store. Initialize space. [13568.336977] Lustre: 428236:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13568.383224] Lustre: sv2857-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13575.279905] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [13591.355594] Lustre: 429176:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [13591.365637] Lustre: 429176:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [13591.370735] Lustre: 429176:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [13591.378651] Lustre: 429176:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [13591.391340] Lustre: 429176:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [13591.399382] Lustre: 429176:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [13599.089689] Lustre: server umount sv2857-OST0000 complete [13614.050021] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13614.072239] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13614.093883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13614.104361] Lustre: Skipped 1 previous similar message [13614.119053] Lustre: Skipped 1 previous similar message [13619.172206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13619.430887] Lustre: server umount lustre-MDT0000 complete [13623.324325] LustreError: 426150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788165761 with bad export cookie 12365591368098275358 [13623.356443] LustreError: 426150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13623.359567] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13623.853088] Lustre: server umount lustre-MDT0001 complete [13636.868664] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [13639.920652] Key type lgssc unregistered [13640.244249] LNet: 430883:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13640.261049] LNetError: 430883:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13640.272108] LNet: Removed LNI 192.168.202.128@tcp [13641.316899] LNet: Removed LNI 192.168.202.128@tcp1 [13642.276442] Key type .llcrypt unregistered [13642.278189] Key type ._llcrypt unregistered [13662.518990] Key type ._llcrypt registered [13662.521443] Key type .llcrypt registered [13662.681271] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [13663.056332] Key type .llcrypt unregistered [13663.060505] Key type ._llcrypt unregistered [13679.048822] Key type ._llcrypt registered [13679.054537] Key type .llcrypt registered [13679.206866] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [13680.555952] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13680.595114] alg: No test for adler32 (adler32-zlib) [13681.745549] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [13682.008468] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [13683.735181] Key type lgssc registered [13685.237372] Lustre: Echo OBD driver; http://www.lustre.org/ [13708.452806] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing lnet_if_list [13709.760835] LNet: Added LNI 192.168.202.128@tcp1 [8/256/0/180] [13711.713634] LNet: Added LNI 192.168.202.128@tcp2 [8/256/0/180] [13723.640613] loop0: detected capacity change from 0 to 204800000 [13723.850410] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13723.862968] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13723.877291] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13741.704624] loop0: detected capacity change from 0 to 204800000 [13742.009795] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13742.029265] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13742.051745] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13761.690084] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13761.701463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13763.024501] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [13763.054949] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13763.271180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13769.606725] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [13782.303108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13782.440677] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13782.504929] Lustre: 434803:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13782.851633] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13788.197640] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [13796.474945] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13803.935734] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13810.283176] loop0: detected capacity change from 0 to 204800000 [13829.740851] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13830.108096] Lustre: sv30299-OST0000: new disk, initializing [13830.139298] Lustre: srv-sv30299-OST0000: No data found on store. Initialize space. [13830.150886] Lustre: 436884:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13830.270640] Lustre: sv30299-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13837.540776] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [13849.142773] loop1: detected capacity change from 0 to 204800000 [13869.169477] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13869.366634] Lustre: sv30299-OST0001: new disk, initializing [13869.370294] Lustre: srv-sv30299-OST0001: No data found on store. Initialize space. [13869.379825] Lustre: 438002:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13869.496817] Lustre: sv30299-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13877.390063] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [13887.141849] Lustre: DEBUG MARKER: SKIP: conf-sanity test_57p need a remote client with no Lustre server role for routing [13888.322859] Lustre: 438749:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [13888.328170] Lustre: 438749:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [13888.333118] Lustre: 438749:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [13888.340691] Lustre: 438749:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [13888.348664] Lustre: 438749:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [13888.355664] Lustre: 438749:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [13890.923673] Lustre: server umount sv30299-OST0001 complete [13904.949684] Lustre: server umount sv30299-OST0000 complete [13919.714508] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13919.724806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13919.752030] Lustre: Skipped 1 previous similar message [13919.768916] Lustre: Skipped 1 previous similar message [13924.834153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13924.856217] Lustre: Skipped 1 previous similar message [13925.945528] Lustre: server umount lustre-MDT0000 complete [13931.098753] LustreError: 434797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788166068 with bad export cookie 16041043474104426892 [13931.104416] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13931.113455] LustreError: 434797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13931.623491] Lustre: server umount lustre-MDT0001 complete [13945.007111] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [13947.905209] Key type lgssc unregistered [13948.340747] LNet: 440757:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13948.346944] LNetError: 440757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13948.355365] LNet: Removed LNI 192.168.202.128@tcp [13949.407180] LNet: Removed LNI 192.168.202.128@tcp1 [13949.411864] LNetError: 440757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13949.424867] LNetError: 440757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13949.458972] LNet: Removed LNI 192.168.202.128@tcp2 [13950.439500] Key type .llcrypt unregistered [13950.442141] Key type ._llcrypt unregistered [13974.333552] Key type ._llcrypt registered [13974.341763] Key type .llcrypt registered [13974.488237] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [13974.953573] Key type .llcrypt unregistered [13974.963074] Key type ._llcrypt unregistered [13995.513553] Key type ._llcrypt registered [13995.515068] Key type .llcrypt registered [13995.608630] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [13996.444142] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13996.465159] alg: No test for adler32 (adler32-zlib) [13997.448987] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [13997.933925] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [13999.719206] Key type lgssc registered [14001.240076] Lustre: Echo OBD driver; http://www.lustre.org/ [14022.913610] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing lnet_if_list [14024.374545] LNet: Added LNI 192.168.202.128@tcp1 [8/256/0/180] [14037.335075] loop0: detected capacity change from 0 to 204800000 [14037.753024] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14037.770502] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14037.790912] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14056.589454] loop0: detected capacity change from 0 to 204800000 [14056.961913] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14056.993811] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14057.043858] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14077.097339] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14077.120941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14078.397556] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [14078.416560] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14078.615812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14083.177745] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [14094.379290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14094.644594] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [14094.721433] Lustre: 444631:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14095.263314] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14103.260996] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [14112.445689] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14118.710657] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14124.520542] loop0: detected capacity change from 0 to 204800000 [14144.652093] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14144.945446] Lustre: sv11988-OST0000: new disk, initializing [14144.949292] Lustre: srv-sv11988-OST0000: No data found on store. Initialize space. [14144.954411] Lustre: 446714:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14144.992964] Lustre: sv11988-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14152.535916] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [14163.820520] loop1: detected capacity change from 0 to 204800000 [14183.480851] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14183.785507] Lustre: sv11988-OST0001: new disk, initializing [14183.796279] Lustre: srv-sv11988-OST0001: No data found on store. Initialize space. [14183.809641] Lustre: 447832:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14183.944548] Lustre: sv11988-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14191.397777] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [14204.540925] LustreError: 448725:0:(nodemap_handler.c:262:nodemap_disc_visibility_update_end_nm()) cfs_fail_timeout id 2404 sleeping for 15000ms [14219.583146] LustreError: 448725:0:(nodemap_handler.c:262:nodemap_disc_visibility_update_end_nm()) cfs_fail_timeout id 2404 awake [14230.660562] Lustre: 449019:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [14230.683756] Lustre: 449019:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [14230.693605] Lustre: 449019:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [14230.705116] Lustre: 449019:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [14230.717142] Lustre: 449019:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [14230.727790] Lustre: 449019:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [14233.034307] Lustre: server umount sv11988-OST0001 complete [14243.712772] Lustre: server umount sv11988-OST0000 complete [14259.168609] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14259.181599] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14259.204170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14260.707023] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14260.724489] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14264.734678] Lustre: server umount lustre-MDT0000 complete [14268.420404] LustreError: 445375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788166406 with bad export cookie 11097279230121518513 [14268.427554] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14268.683578] Lustre: server umount lustre-MDT0001 complete [14281.507235] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [14284.754806] Key type lgssc unregistered [14285.287876] LNet: 451024:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14285.318906] LNetError: 451024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14285.328048] LNet: Removed LNI 192.168.202.128@tcp [14286.379192] LNet: Removed LNI 192.168.202.128@tcp1 [14287.220333] Key type .llcrypt unregistered [14287.221928] Key type ._llcrypt unregistered [14309.863779] Key type ._llcrypt registered [14309.865257] Key type .llcrypt registered [14309.943282] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [14310.165149] Key type .llcrypt unregistered [14310.172242] Key type ._llcrypt unregistered [14329.507506] Key type ._llcrypt registered [14329.509193] Key type .llcrypt registered [14329.717281] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [14330.821688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14330.906713] alg: No test for adler32 (adler32-zlib) [14332.209798] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [14332.467334] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [14334.247181] Key type lgssc registered [14336.038693] Lustre: Echo OBD driver; http://www.lustre.org/ [14361.167655] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing lnet_if_list [14362.714371] LNet: Added LNI 192.168.202.128@tcp1 [8/256/0/180] [14373.911374] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14373.926135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14375.182731] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [14375.210931] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14375.442703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14381.102467] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [14389.869130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14389.961895] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [14390.014304] Lustre: 454295:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14390.475676] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14396.266834] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [14405.317427] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14411.709894] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14416.602733] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing lnet_if_list [14417.118710] Lustre: DEBUG MARKER: rpc test_57r: @@@@@@ FAIL: Can't determine interface name for NID 192.168.202.128@tcp1 [14418.912779] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing check_logdir /tmp/test_logs/1788166553 [14428.130871] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14428.141226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14428.142678] Lustre: Skipped 1 previous similar message [14433.254664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14433.262772] Lustre: Skipped 2 previous similar messages [14433.747943] Lustre: server umount lustre-MDT0000 complete [14437.788567] LustreError: 454287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788166575 with bad export cookie 2795777916378798627 [14437.789382] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14437.795262] LustreError: 454287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14438.149423] Lustre: server umount lustre-MDT0001 complete [14454.885572] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [14458.002891] Key type lgssc unregistered [14458.352459] LNet: 457644:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14458.359325] LNetError: 457644:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14458.366306] LNet: Removed LNI 192.168.202.128@tcp [14459.430627] LNet: Removed LNI 192.168.202.128@tcp1 [14460.582463] Key type .llcrypt unregistered [14460.585053] Key type ._llcrypt unregistered [14483.201297] Key type ._llcrypt registered [14483.204342] Key type .llcrypt registered [14483.370398] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [14484.823597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14484.926172] alg: No test for adler32 (adler32-zlib) [14486.192650] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [14486.527695] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [14488.175178] Key type lgssc registered [14489.709797] Lustre: Echo OBD driver; http://www.lustre.org/ [14496.432347] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing lnet_if_list [14497.483639] LNet: Added LNI 192.168.202.128@tcp1 [8/256/0/180] [14504.851528] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14504.890395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14506.474658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14511.808858] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [14516.704993] LustreError: 459127:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14516.729818] LustreError: 459127:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14521.317023] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14521.805184] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14527.134030] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [14536.006434] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14542.415229] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14550.176283] LNet: Removed LNI 192.168.202.128@tcp1 [14551.186304] Lustre: 460837:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [14551.194216] Lustre: 460837:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [14551.201141] Lustre: 460837:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [14551.210243] Lustre: 460837:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [14551.218920] Lustre: 460837:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [14551.226109] Lustre: 460837:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [14556.816528] Lustre: server umount lustre-MDT0000 complete [14557.664522] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14557.668579] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14557.689531] LustreError: 459126:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14557.695707] Lustre: Skipped 1 previous similar message [14557.720487] LustreError: 459126:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14560.923515] LustreError: 459107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788166698 with bad export cookie 6176687374822757149 [14560.932043] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14560.938229] LustreError: 459107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14563.308771] Lustre: server umount lustre-MDT0001 complete [14575.027283] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [14578.283380] Key type lgssc unregistered [14578.765964] LNet: 461841:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14578.774532] LNetError: 461841:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14578.793778] LNet: Removed LNI 192.168.202.128@tcp [14579.948651] Key type .llcrypt unregistered [14579.950899] Key type ._llcrypt unregistered [14603.857703] Key type ._llcrypt registered [14603.861243] Key type .llcrypt registered [14603.962619] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [14605.240125] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14605.447991] alg: No test for adler32 (adler32-zlib) [14606.560648] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [14606.807620] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [14608.479265] Key type lgssc registered [14609.858246] Lustre: Echo OBD driver; http://www.lustre.org/ [14618.119298] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14618.144418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14619.645818] LNetError: 462468:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.202.128@tcp1 has no available nets [14619.699739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14624.709668] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [14629.866183] LustreError: 463160:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14629.882600] LustreError: 463160:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14632.857261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14633.169800] LNetError: 462468:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.202.128@tcp1 has no available nets [14633.180003] LNetError: 462468:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 2 previous similar messages [14633.250625] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14637.973865] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [14645.667784] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14651.491095] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14660.266183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14660.500964] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [14660.670621] LNetError: 462468:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.202.128@tcp1 has no available nets [14660.683882] LNetError: 462468:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 2 previous similar messages [14660.716985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14667.541911] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [14670.337187] LNetError: 462468:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.202.128@tcp1 has no available nets [14670.353232] LNetError: 462468:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 2 previous similar messages [14675.112454] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14684.641122] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14684.655056] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14684.677664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14686.179941] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14686.183918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14686.190256] Lustre: Skipped 1 previous similar message [14686.205925] Lustre: Skipped 2 previous similar messages [14687.484334] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.28@tcp (stopping) [14689.594584] Lustre: server umount lustre-MDT0000 complete [14691.313202] LustreError: 463159:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14691.332041] LustreError: 463159:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14692.569572] LustreError: 463156:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [14693.453977] LustreError: 463891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788166831 with bad export cookie 792002816025012074 [14693.454847] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14693.478879] LustreError: 463891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14696.419360] LustreError: 463155:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14696.423489] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14696.450573] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14699.754257] Lustre: server umount lustre-MDT0001 complete [14705.900366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [14715.248307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14718.944529] LustreError: 462469:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000000206a8d7 x1875028731799168/t0(0) o250->MGC192.168.202.128@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14719.348634] LNetError: 462468:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.202.128@tcp1 has no available nets [14719.363617] LNetError: 462468:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 3 previous similar messages [14719.389425] LustreError: 464990:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14719.417969] LustreError: 464990:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14719.535669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14719.614369] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [14725.137384] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [14728.678533] LustreError: 466765:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14728.695410] LustreError: 466765:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [14729.710110] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14733.715153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14733.892101] LNetError: 462468:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.202.128@tcp1 has no available nets [14733.905797] LNetError: 462468:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 3 previous similar messages [14733.994116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14739.361864] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [14739.451162] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14748.676377] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14750.852588] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [14757.444102] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14759.147787] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [14765.033983] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14765.047875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14765.049284] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14765.052560] Lustre: Skipped 1 previous similar message [14765.080210] Lustre: Skipped 3 previous similar messages [14770.729296] Lustre: server umount lustre-OST0000 complete [14775.268287] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14775.270434] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14775.291769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14775.295612] Lustre: Skipped 3 previous similar messages [14780.646384] Lustre: server umount lustre-MDT0000 complete [14785.146456] LustreError: 463140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788166923 with bad export cookie 792002816025019788 [14785.149759] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14785.160857] LustreError: 463140:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14785.626157] Lustre: server umount lustre-MDT0001 complete [14796.440120] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [14799.382025] Key type lgssc unregistered [14799.713562] LNet: 469454:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14799.727106] LNetError: 469454:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14799.778470] LNet: Removed LNI 192.168.202.128@tcp [14800.522820] Key type .llcrypt unregistered [14800.524915] Key type ._llcrypt unregistered [14826.326520] Key type ._llcrypt registered [14826.328168] Key type .llcrypt registered [14826.402092] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [14827.582933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14827.639461] alg: No test for adler32 (adler32-zlib) [14828.756241] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [14829.084727] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [14830.743212] Key type lgssc registered [14831.714639] Lustre: Echo OBD driver; http://www.lustre.org/ [14842.261956] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14842.346212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14844.079994] LNetError: 470181:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.202.128@tcp1 has no available nets [14844.131173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14849.848037] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [14853.601817] LustreError: 470874:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14853.615575] LustreError: 470874:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14858.722201] LustreError: 470873:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14859.975757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14860.256248] LNetError: 470181:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.202.128@tcp1 has no available nets [14860.265851] LNetError: 470181:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 3 previous similar messages [14860.443868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14865.835274] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [14873.373565] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14878.041779] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14881.249455] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14881.254459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14881.265303] Lustre: Skipped 1 previous similar message [14881.283502] Lustre: Skipped 1 previous similar message [14885.345648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14885.356716] Lustre: Skipped 1 previous similar message [14887.588888] Lustre: server umount lustre-MDT0000 complete [14890.478990] LustreError: 470873:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14890.509940] LustreError: 470873:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14891.685455] LustreError: 470854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788167029 with bad export cookie 5028764725054728255 [14891.689971] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14891.696655] LustreError: 470854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14892.108136] Lustre: server umount lustre-MDT0001 complete [14908.827732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14909.000750] LNetError: 470181:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.202.128@tcp1 has no available nets [14909.014921] LNetError: 470181:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 3 previous similar messages [14909.149888] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [14909.186729] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14909.437865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14913.922956] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [14923.263389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14923.326543] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [14923.350871] Lustre: 473461:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14923.474362] LNetError: 470181:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.202.128@tcp1 has no available nets [14923.495447] LNetError: 470181:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 1 previous similar message [14923.666500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14927.878506] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [14935.800568] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14941.566803] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14951.347589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14951.639598] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [14951.652743] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [14951.670057] LustreError: 475345:0:(tgt_mount.c:2688:server_fill_super()) Unable to start targets: -2 [14951.675853] LustreError: 475345:0:(tgt_mount.c:2206:server_put_super()) no obd lustre-OST0000 [14951.683485] LustreError: 475345:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [14951.809153] Lustre: server umount lustre-OST0000 complete [14951.815540] LustreError: 475345:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [14963.563313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14963.798296] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [14963.955495] LNetError: 470181:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.202.128@tcp1 has no available nets [14963.960684] LNetError: 470181:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 2 previous similar messages [14963.987265] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14970.986496] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [14973.502217] LNetError: 470181:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.202.128@tcp1 has no available nets [14973.524805] LNetError: 470181:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 1 previous similar message [14973.697567] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [14981.423961] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14990.372309] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14990.504178] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [14990.644186] Lustre: lustre-OST0001: new disk, initializing [14990.654789] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [14990.670230] Lustre: 476888:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14990.750060] LNetError: 470181:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.202.128@tcp1 has no available nets [14990.766325] LNetError: 470181:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 3 previous similar messages [14990.813717] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14998.132553] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [14999.161742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14999.176595] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14999.299915] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [15006.484913] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15014.883338] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15014.885869] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15014.896064] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [15014.915531] Lustre: Skipped 1 previous similar message [15016.374408] Lustre: server umount lustre-OST0001 complete [15020.026110] LustreError: 475812:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15020.057174] LustreError: 475812:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15025.121951] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15025.126337] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15025.132360] LustreError: 475812:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15025.138616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15025.140901] Lustre: Skipped 2 previous similar messages [15026.810190] Lustre: server umount lustre-OST0000 complete [15031.268026] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15031.271126] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15031.289296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15031.295158] Lustre: Skipped 1 previous similar message [15035.364609] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15039.002332] Lustre: server umount lustre-MDT0000 complete [15040.483581] LustreError: 473473:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15040.497958] LustreError: 473473:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [15043.047377] LustreError: 473453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788167180 with bad export cookie 5028764725054728808 [15043.051912] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15043.058702] LustreError: 473453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15043.426115] Lustre: server umount lustre-MDT0001 complete [15055.182564] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [15059.143918] Key type lgssc unregistered [15059.670644] LNet: 478849:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15059.684338] LNetError: 478849:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15060.709387] LNet: Removed LNI 192.168.202.128@tcp [15061.600796] Key type .llcrypt unregistered [15061.604400] Key type ._llcrypt unregistered [15141.744320] Key type ._llcrypt registered [15141.747824] Key type .llcrypt registered [15141.863156] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [15157.366757] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [15158.471666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15158.558341] alg: No test for adler32 (adler32-zlib) [15159.604975] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [15159.862250] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [15161.575166] Key type lgssc registered [15162.669270] Lustre: Echo OBD driver; http://www.lustre.org/ [15201.829984] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15201.843275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15202.970578] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15202.989629] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15203.043733] Lustre: lustre-MDT0000: new disk, initializing [15203.106870] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15203.118860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15208.738560] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [15220.255857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15220.379745] Lustre: 483786:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15220.427988] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15220.430604] Lustre: Skipped 1 previous similar message [15220.494699] Lustre: lustre-MDT0001: new disk, initializing [15220.579133] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15220.616468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15220.628445] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15225.548902] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [15236.405337] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15242.906601] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15249.704152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15250.026064] Lustre: lustre-OST0000: new disk, initializing [15250.028364] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15250.033208] Lustre: 485753:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15250.137752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15256.442397] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [15257.721823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15257.734561] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15257.830967] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15266.999218] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15272.945783] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15273.209142] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15278.449823] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15278.696286] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15283.680921] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15283.690722] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15283.699055] Lustre: Skipped 1 previous similar message [15283.699318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15283.721047] LustreError: Skipped 1 previous similar message [15287.359855] Lustre: server umount lustre-OST0000 complete [15292.388694] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15292.396325] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15292.412125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15292.420482] Lustre: Skipped 1 previous similar message [15293.923567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15293.926938] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15297.580677] Lustre: server umount lustre-MDT0000 complete [15301.137815] LustreError: 483779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788167439 with bad export cookie 9767412996287283760 [15301.151615] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15301.153244] LustreError: 483779:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15301.571840] Lustre: server umount lustre-MDT0001 complete [15310.785423] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 05:10:46 (1788167446) [15319.215246] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 05:10:55 (1788167455) [15328.923104] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15329.490755] LustreError: 488505:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15329.510633] LustreError: 488505:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [15329.552963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15334.307136] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [15334.884698] LustreError: 488506:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15340.002702] LustreError: 488505:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15343.153361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15343.422487] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15348.166498] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [15355.714646] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15361.076988] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15368.179452] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15368.493504] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15375.100650] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [15382.429746] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15390.438923] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [15392.396873] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [15394.427865] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [15396.697731] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [15398.703643] Lustre: Failing over lustre-MDT0000 [15398.969172] Lustre: server umount lustre-MDT0000 complete [15399.672074] LustreError: 488804:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [15399.911246] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15399.924618] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15404.766983] LustreError: 488501:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [15404.791744] LustreError: 488501:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [15412.955213] LustreError: 488500:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [15412.972659] LustreError: 488500:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [15416.801802] Lustre: 482010:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788167538/real 1788167538] req@00000000b4333db0 x1875029311690112/t0(0) o400->MGC192.168.202.128@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788167554 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15416.849147] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15419.297413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15427.041150] LustreError: 482009:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000061f9dbf x1875029311695360/t0(0) o250->MGC192.168.202.128@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15427.336945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15427.374476] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [15431.398406] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [15432.104413] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [15432.688207] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15432.714960] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [15432.791058] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [15441.962666] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15443.828648] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [15447.159715] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [15448.930315] Lustre: Failing over lustre-MDT0000 [15449.268936] Lustre: server umount lustre-MDT0000 complete [15451.892220] LustreError: 488500:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [15451.914338] LustreError: 488500:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [15453.152615] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15453.154955] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15453.177784] Lustre: Skipped 4 previous similar messages [15467.621198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15467.717417] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15467.984789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15468.033859] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [15468.292832] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [15471.982061] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [15473.131305] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [15473.137735] Lustre: Skipped 2 previous similar messages [15473.161948] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [15473.194268] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [15481.426366] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15482.822040] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [15488.483130] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15488.496202] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15488.498535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15488.498546] Lustre: Skipped 1 previous similar message [15488.500547] Lustre: Skipped 1 previous similar message [15493.601164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15493.608613] Lustre: Skipped 4 previous similar messages [15494.311544] Lustre: server umount lustre-MDT0000 complete [15498.404262] LustreError: 488487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788167636 with bad export cookie 9767412996287286469 [15498.427556] LustreError: 488487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15498.431258] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15498.729290] LustreError: 488501:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15498.743995] LustreError: 488501:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [15498.880901] Lustre: server umount lustre-MDT0001 complete [15504.965979] Lustre: server umount lustre-OST0000 complete [15520.336951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15524.651563] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [15533.907231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15534.294963] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15534.297217] Lustre: Skipped 1 previous similar message [15540.133867] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [15549.386450] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15556.409150] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15563.442283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15565.697773] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [15568.484297] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [15576.709514] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15586.273538] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15586.288244] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15586.297795] Lustre: Skipped 2 previous similar messages [15586.305911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15592.278337] Lustre: server umount lustre-OST0000 complete [15603.685711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15603.689453] Lustre: Skipped 6 previous similar messages [15610.336468] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15610.644783] Lustre: server umount lustre-MDT0000 complete [15613.923774] LustreError: 494840:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15613.947162] LustreError: 494840:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [15614.902513] LustreError: 494824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788167752 with bad export cookie 9767412996287287211 [15614.910402] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15614.924116] LustreError: 494824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15615.317108] Lustre: server umount lustre-MDT0001 complete [15628.279826] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [15631.536442] Key type lgssc unregistered [15631.828272] LNet: 498379:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15631.836479] LNetError: 498379:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15631.848204] LNet: Removed LNI 192.168.202.128@tcp [15632.794547] Key type .llcrypt unregistered [15632.796478] Key type ._llcrypt unregistered [15659.617708] Key type ._llcrypt registered [15659.620345] Key type .llcrypt registered [15659.735386] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [15660.842423] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15660.893032] alg: No test for adler32 (adler32-zlib) [15661.922130] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [15662.159349] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [15663.847347] Key type lgssc registered [15664.918768] Lustre: Echo OBD driver; http://www.lustre.org/ [15673.019389] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15673.029309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15674.418803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15679.601445] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [15684.581772] LustreError: 499896:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15684.605073] LustreError: 499896:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [15688.233906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15688.643259] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15693.031549] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [15700.546641] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15705.820688] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15712.371620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15712.723153] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15718.596326] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [15719.936058] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [15724.957945] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15735.270129] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15735.286051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15735.303132] Lustre: Skipped 2 previous similar messages [15740.391734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15740.398968] Lustre: Skipped 2 previous similar messages [15741.788818] Lustre: server umount lustre-MDT0000 complete [15745.509419] LustreError: 500919:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15745.526442] LustreError: 500919:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [15745.580259] LustreError: 499877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788167883 with bad export cookie 9806857877315247676 [15745.597832] LustreError: 499877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15745.598537] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15745.955235] Lustre: server umount lustre-MDT0001 complete [15760.131685] Lustre: server umount lustre-OST0000 complete [15774.164383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15774.799729] LustreError: 503607:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15774.879990] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15779.384730] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [15780.320873] LustreError: 503608:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15784.423588] LustreError: 503607:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15788.265386] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15788.770797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15793.256980] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [15801.073496] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15806.933617] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15812.938349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15813.356513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15819.524144] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [15819.839191] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [15827.084087] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15840.231423] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15840.234199] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15840.240259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15840.253099] Lustre: Skipped 3 previous similar messages [15843.477390] Lustre: server umount lustre-MDT0000 complete [15845.345642] LustreError: 504336:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15845.379501] LustreError: 504336:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [15846.709655] LustreError: 503589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788167984 with bad export cookie 9806857877315248789 [15846.715245] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15846.717475] LustreError: 503589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15847.094244] Lustre: server umount lustre-MDT0001 complete [15860.668435] Lustre: server umount lustre-OST0000 complete [15874.099257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15874.424153] LustreError: 507518:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15874.522545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15878.855676] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [15887.768065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15892.817198] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [15900.885568] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15906.697733] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15913.550331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15914.097474] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15914.101782] Lustre: Skipped 1 previous similar message [15920.124595] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [15921.393718] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [15930.540658] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15939.556498] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15939.565870] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15939.573459] Lustre: Skipped 1 previous similar message [15939.580579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15939.593779] Lustre: Skipped 2 previous similar messages [15942.590810] Lustre: server umount lustre-OST0000 complete [15949.799744] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15949.809516] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15949.824837] Lustre: Skipped 1 previous similar message [15949.830566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15949.838772] Lustre: Skipped 2 previous similar messages [15953.003177] Lustre: server umount lustre-MDT0000 complete [15955.938413] LustreError: 508760:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15955.968956] LustreError: 508760:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [15956.295861] LustreError: 507498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788168094 with bad export cookie 9806857877315249804 [15956.305876] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15956.313020] LustreError: 507498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15956.699916] Lustre: server umount lustre-MDT0001 complete [15967.341660] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [15969.737222] Key type lgssc unregistered [15970.057258] LNet: 511052:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15970.070350] LNetError: 511052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15971.110410] LNet: Removed LNI 192.168.202.128@tcp [15972.052148] Key type .llcrypt unregistered [15972.054473] Key type ._llcrypt unregistered [15995.236923] Key type ._llcrypt registered [15995.239134] Key type .llcrypt registered [15995.342819] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [15996.815512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15996.847082] alg: No test for adler32 (adler32-zlib) [15997.993521] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [15998.274847] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [15999.969798] Key type lgssc registered [16001.135710] Lustre: Echo OBD driver; http://www.lustre.org/ [16008.084194] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16008.103464] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16008.109432] LustreError: 512492:0:(osd_handler.c:8554:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [16008.123349] LustreError: 512492:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [16008.133536] LustreError: 512492:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [16008.144770] LustreError: 512492:0:(tgt_mount.c:2656:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [16008.159078] LustreError: 512492:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [16014.723298] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [16014.737030] LustreError: 512890:0:(osd_handler.c:8554:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [16014.759508] LustreError: 512890:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [16014.768454] LustreError: 512890:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [16014.780509] LustreError: 512890:0:(tgt_mount.c:2656:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [16014.791910] LustreError: 512890:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [16035.840745] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [16039.978603] Key type lgssc unregistered [16040.315804] LNet: 513895:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16040.321952] LNetError: 513895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16040.340474] LNet: Removed LNI 192.168.202.128@tcp [16041.342646] Key type .llcrypt unregistered [16041.344055] Key type ._llcrypt unregistered [16059.468237] Key type ._llcrypt registered [16059.473360] Key type .llcrypt registered [16059.601259] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [16072.776742] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [16073.694801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16073.744194] alg: No test for adler32 (adler32-zlib) [16074.870775] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [16075.149152] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [16076.816943] Key type lgssc registered [16077.685443] Lustre: Echo OBD driver; http://www.lustre.org/ [16113.946358] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16113.969900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16115.155941] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16115.206242] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16115.272126] Lustre: lustre-MDT0000: new disk, initializing [16115.330912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16115.351349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16118.994528] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [16132.736820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16132.890954] Lustre: 517033:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16132.975519] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16132.981831] Lustre: Skipped 1 previous similar message [16133.053073] Lustre: lustre-MDT0001: new disk, initializing [16133.146871] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16133.191905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16133.204823] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16138.470975] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [16148.830736] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16153.737630] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16160.380109] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16160.598817] Lustre: lustre-OST0000: new disk, initializing [16160.603362] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16160.609312] Lustre: 519000:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16160.687643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16166.568386] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [16167.965047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16167.970554] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16168.009274] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16173.562930] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16177.611560] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16177.750973] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16183.724193] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16184.106851] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16188.388435] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16188.396057] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16188.411770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16188.428819] Lustre: Skipped 1 previous similar message [16192.440272] Lustre: server umount lustre-OST0000 complete [16198.630320] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16198.637960] Lustre: Skipped 1 previous similar message [16198.641762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16203.539432] Lustre: server umount lustre-MDT0000 complete [16207.525029] LustreError: 517024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788168345 with bad export cookie 9664422855067555469 [16207.526590] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16207.538866] LustreError: 517024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16207.798914] Lustre: server umount lustre-MDT0001 complete [16217.659117] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [16219.685876] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 05:25:55 (1788168355) [16226.868427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16227.491507] LustreError: 521461:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16227.511409] LustreError: 521461:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [16227.565193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16232.755519] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [16232.937619] LustreError: 521462:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16238.059189] LustreError: 521461:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16241.663957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16242.035620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16246.488157] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [16254.181175] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16260.255206] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16266.608714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16267.051230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16273.062527] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [16280.844727] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16287.292293] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16287.418169] Lustre: lustre-OST0001: new disk, initializing [16287.421862] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [16287.429353] Lustre: 524297:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16287.488856] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [16295.665030] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [16297.086117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [16297.101353] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [16297.210162] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [16306.663446] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16312.817546] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16312.820891] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16312.821707] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [16312.821712] Lustre: Skipped 1 previous similar message [16312.836558] Lustre: Skipped 2 previous similar messages [16312.851427] LustreError: Skipped 1 previous similar message [16316.769500] Lustre: server umount lustre-OST0001 complete [16317.927497] LustreError: 523281:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16317.948461] LustreError: 523281:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16323.058168] LustreError: 523283:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16323.086123] LustreError: 523283:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16333.287347] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16333.290197] LustreError: 523283:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16333.309293] Lustre: Skipped 1 previous similar message [16333.311990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16333.321014] LustreError: 523283:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [16333.339186] Lustre: Skipped 3 previous similar messages [16338.402508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16338.406975] Lustre: Skipped 1 previous similar message [16347.103114] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16347.236674] Lustre: server umount lustre-OST0000 complete [16353.766781] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16353.782679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16353.785034] Lustre: Skipped 2 previous similar messages [16357.494373] Lustre: server umount lustre-MDT0000 complete [16358.883873] LustreError: 521458:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16358.908414] LustreError: 521458:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [16361.223482] LustreError: 521442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788168499 with bad export cookie 9664422855067556344 [16361.227329] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16367.687442] Lustre: server umount lustre-MDT0001 complete [16379.295878] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [16382.096436] Key type lgssc unregistered [16382.483914] LNet: 526287:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16382.503478] LNetError: 526287:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16382.531671] LNet: Removed LNI 192.168.202.128@tcp [16383.795242] Key type .llcrypt unregistered [16383.798772] Key type ._llcrypt unregistered [16441.989210] Key type ._llcrypt registered [16441.994605] Key type .llcrypt registered [16442.198259] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [16443.639280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16443.682218] alg: No test for adler32 (adler32-zlib) [16444.661287] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [16444.965278] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [16446.671784] Key type lgssc registered [16448.293666] Lustre: Echo OBD driver; http://www.lustre.org/ [16457.292408] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16457.309968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16458.532463] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [16458.593244] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16458.818941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16464.199673] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [16474.209923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16474.403773] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [16474.475268] Lustre: 529090:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16474.796594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16479.578786] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [16488.513244] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16495.034374] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16500.707547] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16500.709121] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16500.724473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16503.023741] Lustre: server umount lustre-MDT0000 complete [16507.201988] LustreError: 529081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788168645 with bad export cookie 251064656242385082 [16507.210108] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16507.225867] LustreError: 529081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16507.726257] Lustre: server umount lustre-MDT0001 complete [16517.685972] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 05:30:53 (1788168653) [16525.473656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16526.000205] LustreError: 531601:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16526.018809] LustreError: 531601:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [16526.074064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16530.891135] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [16531.432781] LustreError: 531602:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16531.450294] LustreError: 531602:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16536.547205] LustreError: 531601:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16538.917213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16539.219091] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16543.454316] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [16551.270100] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16556.991530] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16562.923748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16563.078830] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [16563.262453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16568.613129] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [16575.176539] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16581.877467] Lustre: Permanently deactivating lustre-OST0000 [16581.888604] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [16587.778959] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [16588.936557] LustreError: 534543:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [16588.946934] LustreError: 534543:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [16588.953397] LustreError: 534543:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [16591.991163] LustreError: 534592:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [16591.995915] LustreError: 534592:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [16600.242916] Lustre: server umount lustre-OST0000 complete [16602.661864] LustreError: 534841:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [16602.667163] LustreError: 534841:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [16604.649474] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16604.666356] Lustre: Skipped 1 previous similar message [16604.686920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16604.690201] Lustre: Skipped 2 previous similar messages [16609.760617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16609.775203] Lustre: Skipped 1 previous similar message [16614.882427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16614.888572] Lustre: Skipped 1 previous similar message [16614.954446] Lustre: server umount lustre-MDT0000 complete [16618.591930] LustreError: 534296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788168756 with bad export cookie 251064656242385663 [16618.595678] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16618.599392] LustreError: 534296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16628.348651] Lustre: server umount lustre-MDT0001 complete [16637.622534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16642.581309] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [16645.206736] LustreError: 536003:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [16652.235968] Lustre: 536342:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.128@tcp [16654.526105] Lustre: server umount MGS complete [16665.896325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16666.314795] LustreError: 537065:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16666.347519] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [16666.355655] Lustre: Skipped 1 previous similar message [16666.466886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16671.549567] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [16671.713759] LustreError: 537066:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16680.020167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16680.241495] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [16683.736646] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [16690.563716] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16697.980573] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16700.412665] Lustre: Permanently reactivating lustre-OST0000 [16700.422352] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [16700.426152] Lustre: Skipped 2 previous similar messages [16730.079174] LustreError: 538859:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [16737.588375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16738.058768] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16738.062272] Lustre: Skipped 1 previous similar message [16744.481850] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [16751.702996] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16758.567891] Lustre: server umount lustre-OST0000 complete [16762.338301] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16762.352231] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16762.368702] Lustre: Skipped 1 previous similar message [16762.382680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16765.921630] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16771.041959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16771.050662] Lustre: Skipped 3 previous similar messages [16772.136289] Lustre: server umount lustre-MDT0000 complete [16775.145712] Lustre: 528413:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788168897/real 1788168897] req@00000000b1defaf0 x1875030659135744/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788168913 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16775.173887] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16775.201825] LustreError: 538161:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16775.219687] LustreError: 538161:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16776.180834] Lustre: 528410:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788168898/real 1788168898] req@00000000b1c92acf x1875030659136512/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788168914 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16776.240725] LustreError: 539789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788168914 with bad export cookie 251064656242386713 [16776.244606] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16776.251944] LustreError: 539789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [16776.618466] Lustre: server umount lustre-MDT0001 complete [16788.292734] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [16791.526409] Key type lgssc unregistered [16791.800465] LNet: 541475:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16791.808632] LNetError: 541475:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16791.823686] LNet: Removed LNI 192.168.202.128@tcp [16792.619271] Key type .llcrypt unregistered [16792.621234] Key type ._llcrypt unregistered [16811.412083] Key type ._llcrypt registered [16811.413653] Key type .llcrypt registered [16811.544124] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [16826.964215] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [16828.002082] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16828.053247] alg: No test for adler32 (adler32-zlib) [16829.334065] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [16829.647277] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [16831.311169] Key type lgssc registered [16832.429209] Lustre: Echo OBD driver; http://www.lustre.org/ [16872.861517] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16872.891093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16874.057396] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16874.084982] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16874.145922] Lustre: lustre-MDT0000: new disk, initializing [16874.218840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16874.234776] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16879.308349] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [16890.233287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16890.288595] Lustre: 544633:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16890.339857] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16890.343407] Lustre: Skipped 1 previous similar message [16890.432724] Lustre: lustre-MDT0001: new disk, initializing [16890.489105] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16890.517979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16890.525871] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16895.305097] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [16905.017779] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16910.635882] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16919.999223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16920.365555] Lustre: lustre-OST0000: new disk, initializing [16920.372612] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16920.381141] Lustre: 546600:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16920.469583] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16926.222651] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [16930.325522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16930.344427] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16930.470189] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16936.420406] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16942.623855] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16942.907682] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16947.775533] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16947.934329] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16950.753542] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16950.765772] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16950.815636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16951.275380] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16951.285271] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16956.388851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16956.403141] Lustre: Skipped 2 previous similar messages [16956.789647] Lustre: server umount lustre-OST0000 complete [16962.536814] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16962.546972] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16962.566924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16962.581789] Lustre: Skipped 1 previous similar message [16966.632717] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16966.640518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16966.661287] Lustre: Skipped 1 previous similar message [16971.747559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16971.763647] Lustre: Skipped 1 previous similar message [16976.351161] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16976.665824] Lustre: server umount lustre-MDT0000 complete [16981.994890] LustreError: 544644:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16982.020043] LustreError: 544644:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [16982.296221] LustreError: 544624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788169120 with bad export cookie 12095417740115272326 [16982.306483] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16982.313548] LustreError: 544624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16982.824591] Lustre: server umount lustre-MDT0001 complete [16993.609915] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [16995.413470] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 05:38:51 (1788169131) [17002.809040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17003.270531] LustreError: 549063:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17003.367513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17008.616938] LustreError: 549064:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17008.756485] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [17013.729522] LustreError: 549063:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17017.343336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17017.761180] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17021.888562] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [17032.598385] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [17041.072631] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [17051.571272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17052.470837] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17060.613291] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [17067.490992] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17069.261963] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [17076.227694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [17083.323586] Lustre: server umount lustre-OST0000 complete [17089.504285] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [17089.517204] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17089.530528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17089.541362] Lustre: Skipped 1 previous similar message [17089.576629] Lustre: Skipped 1 previous similar message [17099.744374] Lustre: 542856:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788169221/real 1788169221] req@000000007b19e06d x1875031062813440/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788169237 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [17099.770994] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17100.767477] Lustre: 542854:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788169222/real 1788169222] req@000000001a29a6d1 x1875031062814208/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788169238 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [17101.280894] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [17101.500412] Lustre: server umount lustre-MDT0000 complete [17104.864497] LustreError: 549064:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17104.883951] LustreError: 549064:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [17105.465640] LustreError: 549045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788169243 with bad export cookie 12095417740115273208 [17105.466563] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17105.471342] LustreError: 549045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [17105.718448] Lustre: server umount lustre-MDT0001 complete [17120.734258] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [17123.461727] Key type lgssc unregistered [17123.743799] LNet: 552791:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17123.747524] LNetError: 552791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17123.763608] LNet: Removed LNI 192.168.202.128@tcp [17124.343344] Key type .llcrypt unregistered [17124.344868] Key type ._llcrypt unregistered [17152.466406] Key type ._llcrypt registered [17152.469303] Key type .llcrypt registered [17152.641062] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [17153.131308] Key type .llcrypt unregistered [17153.132827] Key type ._llcrypt unregistered [17174.917282] Key type ._llcrypt registered [17174.922183] Key type .llcrypt registered [17175.049991] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [17176.224679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [17176.396415] alg: No test for adler32 (adler32-zlib) [17177.657489] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [17178.119944] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [17179.920295] Key type lgssc registered [17181.396990] Lustre: Echo OBD driver; http://www.lustre.org/ [17191.131191] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [17191.170863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17192.992716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17198.053079] LustreError: 555005:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17198.072160] LustreError: 555005:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [17198.321458] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [17202.148860] LustreError: 555006:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17207.275819] LustreError: 555005:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17207.844955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17208.200381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17213.411896] LustreError: 555006:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17213.424694] LustreError: 555006:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [17215.512548] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [17217.516658] LustreError: 555000:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17217.541773] LustreError: 555000:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [17217.542639] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [17224.148232] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17227.763972] LustreError: 555000:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17227.796570] LustreError: 555000:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [17230.657236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17230.952593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17235.596712] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [17246.181497] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [17246.197098] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17246.216525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17247.109815] Lustre: server umount lustre-OST0000 complete [17251.305436] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [17251.311396] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17251.319545] Lustre: Skipped 1 previous similar message [17251.323275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17251.330161] Lustre: Skipped 1 previous similar message [17253.567378] Lustre: server umount lustre-MDT0000 complete [17256.930700] LustreError: 555006:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17256.941608] LustreError: 555006:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [17257.446143] LustreError: 554986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788169395 with bad export cookie 9742713200143193165 [17257.453652] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17257.458707] LustreError: 554986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [17257.846817] Lustre: server umount lustre-MDT0001 complete [17270.107230] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [17272.902749] Key type lgssc unregistered [17273.211537] LNet: 558155:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17273.216482] LNetError: 558155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17273.237410] LNet: Removed LNI 192.168.202.128@tcp [17274.159490] Key type .llcrypt unregistered [17274.161540] Key type ._llcrypt unregistered [17302.545892] Key type ._llcrypt registered [17302.548917] Key type .llcrypt registered [17302.657829] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [17303.810798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [17303.846267] alg: No test for adler32 (adler32-zlib) [17304.910834] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [17305.171039] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [17306.839299] Key type lgssc registered [17308.194877] Lustre: Echo OBD driver; http://www.lustre.org/ [17315.733392] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [17315.744358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17331.295453] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [17336.416233] LustreError: 559546:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.128@tcp: failed processing log, type 4: rc = -110 [17363.104516] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17371.130582] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [17381.490562] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17389.279333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17389.732108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17389.805683] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [17394.840066] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [17395.173772] LustreError: 559571:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17395.194662] LustreError: 559571:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [17397.226317] LustreError: 560582:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17397.246264] LustreError: 560582:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [17402.342653] LustreError: 560584:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17402.366169] LustreError: 560584:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [17404.898655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17405.515134] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17410.736967] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [17418.898280] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [17425.177135] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [17435.003492] Lustre: server umount lustre-OST0000 complete [17439.727784] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [17439.742099] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17439.756819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17441.249282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17441.251531] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17446.382204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17446.389133] Lustre: Skipped 2 previous similar messages [17448.676500] Lustre: server umount lustre-MDT0000 complete [17451.490964] LustreError: 560587:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17451.511026] LustreError: 560587:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [17452.511205] Lustre: 558882:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788169574/real 1788169574] req@00000000c28cc8ee x1875031561894016/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788169590 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [17452.538223] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17452.977966] LustreError: 559553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788169590 with bad export cookie 3567306524999501540 [17452.981965] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17452.991603] LustreError: 559553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [17453.516713] Lustre: server umount lustre-MDT0001 complete [17464.060618] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [17467.159101] Key type lgssc unregistered [17467.443517] LNet: 563117:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17467.450140] LNetError: 563117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17467.482683] LNet: Removed LNI 192.168.202.128@tcp [17468.386241] Key type .llcrypt unregistered [17468.389298] Key type ._llcrypt unregistered [17491.865602] Key type ._llcrypt registered [17491.867843] Key type .llcrypt registered [17492.009602] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [17493.379558] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [17493.530025] alg: No test for adler32 (adler32-zlib) [17494.673489] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [17494.909466] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [17496.607184] Key type lgssc registered [17497.890122] Lustre: Echo OBD driver; http://www.lustre.org/ [17505.538414] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [17505.561992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17506.988429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17512.031596] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [17516.512947] LustreError: 564534:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17516.538289] LustreError: 564534:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [17521.226603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17521.534913] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17525.636530] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [17533.703828] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [17539.845566] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [17546.777378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17547.156854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17551.289635] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [17554.144684] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [17562.346456] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17566.690209] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17566.691329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17566.698170] Lustre: Skipped 2 previous similar messages [17566.721620] Lustre: Skipped 2 previous similar messages [17567.715547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17567.732843] Lustre: Skipped 2 previous similar messages [17569.529038] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.28@tcp (stopping) [17572.560926] Lustre: server umount lustre-MDT0000 complete [17572.839881] LustreError: 564535:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17572.868557] LustreError: 564535:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [17574.621755] LustreError: 564529:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [17575.172984] LustreError: 564516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.28@tcp arrived at 1788169713 with bad export cookie 13852563702624471375 [17575.192836] LustreError: 564516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [17577.479674] LustreError: 564515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788169715 with bad export cookie 13852563702624470486 [17577.485116] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17577.486734] LustreError: 564515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [17577.953458] LustreError: 564534:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17577.954229] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17577.966287] LustreError: 564534:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [17577.980576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17583.073162] LustreError: 564535:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17583.076052] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17583.094214] LustreError: 564535:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [17583.690935] Lustre: server umount lustre-OST0000 complete [17590.307272] LustreError: 565265:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000dfe3ebbb x1875031760075904/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [17590.550355] Lustre: server umount lustre-MDT0001 complete [17601.881863] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [17604.753960] Key type lgssc unregistered [17604.953795] LNet: 568220:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17604.964817] LNetError: 568220:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17606.002363] LNet: Removed LNI 192.168.202.128@tcp [17606.933243] Key type .llcrypt unregistered [17606.935477] Key type ._llcrypt unregistered [17635.579761] Key type ._llcrypt registered [17635.580956] Key type .llcrypt registered [17635.673704] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [17636.833725] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [17636.914837] alg: No test for adler32 (adler32-zlib) [17637.980315] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [17638.254358] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [17639.919180] Key type lgssc registered [17641.332659] Lustre: Echo OBD driver; http://www.lustre.org/ [17650.651415] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [17650.710135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17652.577927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17658.468464] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [17662.947237] LustreError: 569637:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17662.968935] LustreError: 569637:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [17668.072837] LustreError: 569638:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17668.374621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17668.969191] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17674.276517] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [17683.980386] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [17690.866843] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [17698.033392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17698.385581] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17706.741224] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [17715.745147] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17723.027951] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:193) [17730.527893] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [17730.532189] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17730.539820] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [17732.856700] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.28@tcp (stopping) [17733.608148] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17736.773863] Lustre: server umount lustre-MDT0001 complete [17737.949406] LustreError: 570619:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [17743.842922] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [17743.865616] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17743.872591] LustreError: 569638:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17743.883263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17743.883270] Lustre: Skipped 2 previous similar messages [17743.897920] LustreError: 569638:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [17748.683701] Lustre: server umount lustre-OST0000 complete [17748.960728] LustreError: 569637:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17753.642114] Lustre: server umount lustre-MDT0000 complete [17770.798934] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [17774.152592] Key type lgssc unregistered [17774.604649] LNet: 573346:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17774.622671] LNetError: 573346:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17774.654123] LNet: Removed LNI 192.168.202.128@tcp [17775.772353] Key type .llcrypt unregistered [17775.777489] Key type ._llcrypt unregistered [17806.708601] Key type ._llcrypt registered [17806.710679] Key type .llcrypt registered [17806.801746] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [17824.455362] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [17826.113758] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [17826.320871] alg: No test for adler32 (adler32-zlib) [17827.730077] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [17828.102430] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [17829.807597] Key type lgssc registered [17831.503545] Lustre: Echo OBD driver; http://www.lustre.org/ [17880.555102] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [17899.108883] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [17899.141922] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17900.437538] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [17900.491098] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [17900.596894] Lustre: lustre-MDT0000: new disk, initializing [17900.749565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17900.804155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [17905.847832] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [17922.748345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17923.114511] Lustre: lustre-OST0000: new disk, initializing [17923.120619] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [17923.123260] Lustre: Skipped 1 previous similar message [17923.147319] Lustre: 578388:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [17923.254265] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17930.540251] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [17932.864688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [17932.874704] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [17932.924598] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [17940.966531] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17956.952591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17957.146684] Lustre: 577368:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [17957.219643] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [17957.339945] Lustre: lustre-MDT0001: new disk, initializing [17957.505942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17957.562062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [17957.581890] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [17962.601932] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [17973.286977] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [17973.587696] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [17979.252751] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [17979.524184] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [17984.755803] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.28@tcp (stopping) [17985.516364] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17985.525951] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [17985.539129] Lustre: Skipped 1 previous similar message [17985.570532] Lustre: Skipped 1 previous similar message [17989.872442] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.28@tcp (stopping) [17990.831668] Lustre: server umount lustre-MDT0001 complete [17995.755159] LustreError: 577380:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17995.779143] LustreError: 577380:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [18000.866688] LustreError: 577376:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18000.885242] LustreError: 577376:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [18005.994839] LustreError: 577381:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18006.039257] LustreError: 577381:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [18011.105877] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18011.107192] LustreError: 577380:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18011.135470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [18015.938615] Lustre: server umount lustre-OST0000 complete [18016.266310] LustreError: 577381:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18027.740625] Lustre: server umount lustre-MDT0000 complete [18042.397919] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [18044.842913] Key type lgssc unregistered [18045.222938] LNet: 582133:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [18045.227662] LNetError: 582133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [18045.239419] LNet: Removed LNI 192.168.202.128@tcp [18046.072745] Key type .llcrypt unregistered [18046.075443] Key type ._llcrypt unregistered [18088.523356] Key type ._llcrypt registered [18088.525749] Key type .llcrypt registered [18088.643785] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [18088.835344] Key type .llcrypt unregistered [18088.836742] Key type ._llcrypt unregistered [18109.448345] Key type ._llcrypt registered [18109.455892] Key type .llcrypt registered [18109.635745] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [18110.831885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [18111.059525] alg: No test for adler32 (adler32-zlib) [18112.168929] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [18112.457625] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [18114.111254] Key type lgssc registered [18115.160860] Lustre: Echo OBD driver; http://www.lustre.org/ [18147.945784] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [18147.983028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18149.414548] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [18149.507805] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [18149.588705] Lustre: lustre-MDT0000: new disk, initializing [18149.644237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [18149.665242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [18155.187919] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [18166.923631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18167.084328] Lustre: 585515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [18167.132594] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [18167.136677] Lustre: Skipped 1 previous similar message [18167.190963] Lustre: lustre-MDT0001: new disk, initializing [18167.272905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [18167.318063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [18167.330147] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [18172.154841] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [18183.514624] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [18189.689285] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [18197.505517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [18197.847662] Lustre: lustre-OST0000: new disk, initializing [18197.850726] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [18197.863096] Lustre: 587482:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [18197.994351] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [18203.734179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [18203.747820] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [18203.962915] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [18206.037970] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [18216.466594] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [18223.584277] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [18223.588199] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18223.607099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [18224.618383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [18224.629095] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18224.645104] Lustre: Skipped 1 previous similar message [18229.449287] Lustre: server umount lustre-MDT0000 complete [18233.528948] LustreError: 587481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788170371 with bad export cookie 6140763917994255673 [18233.540190] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [18233.545055] LustreError: 587481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [18233.996880] Lustre: server umount lustre-MDT0001 complete [18247.498747] Lustre: server umount lustre-OST0000 complete [18259.683213] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 05:59:55 (1788170395) [18268.610239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18269.606994] LustreError: 589510:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18269.634740] LustreError: 589510:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [18269.791437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [18274.786520] LustreError: 589511:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18275.698883] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [18279.918562] LustreError: 589510:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18285.032612] LustreError: 589511:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18285.960579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18286.365875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [18292.408658] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [18301.695141] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [18307.089359] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [18326.887344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [18327.080573] Lustre: Found index 0 for lustre-OST0000, updating log [18327.263364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [18333.237705] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [18340.600547] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [18340.644577] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [18345.952252] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [18345.954807] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18345.998960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [18346.006556] Lustre: Skipped 2 previous similar messages [18346.948986] Lustre: server umount lustre-OST0000 complete [18363.360771] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [18363.372032] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18363.391578] Lustre: Skipped 1 previous similar message [18363.401338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [18367.458221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [18367.458527] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18368.848741] Lustre: server umount lustre-MDT0000 complete [18372.576103] LustreError: 589507:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18372.626530] LustreError: 589507:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [18374.080409] LustreError: 589491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788170511 with bad export cookie 6140763917994257066 [18374.089491] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [18374.100623] LustreError: 589491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [18374.817551] Lustre: server umount lustre-MDT0001 complete [18384.712332] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 06:02:00 (1788170520) [18390.047300] LNet: 1 local NIs in recovery (showing 1): 192.168.202.128@tcp [19011.159562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19011.629470] LustreError: 595930:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19011.734522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [19016.290950] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [19017.189733] LustreError: 595931:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19021.288619] LustreError: 595930:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19025.321954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19025.790820] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [19030.184959] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [19038.439980] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [19044.793852] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [19065.279538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [19065.609790] Lustre: Found index 0 for lustre-OST0000, updating log [19066.035287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [19067.936233] LNet: 1 local NIs in recovery (showing 1): 192.168.202.128@tcp [19073.298620] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [19081.242700] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [19083.744803] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [19083.765474] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19083.788094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [19083.800305] Lustre: Skipped 1 previous similar message [19089.792476] Lustre: server umount lustre-OST0000 complete [19094.574261] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19094.587964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [19094.597161] Lustre: Skipped 1 previous similar message [19096.673415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [19096.684353] Lustre: Skipped 4 previous similar messages [19099.927583] Lustre: server umount lustre-MDT0000 complete [19100.840493] LustreError: 595930:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19100.868789] LustreError: 595930:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [19103.872413] LustreError: 595911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788171241 with bad export cookie 6140763917994258151 [19103.882552] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [19103.890291] LustreError: 595911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [19103.910651] LustreError: 584043:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000bcf57189 x1875032408197760/t0(0) o8->lustre-OST0000-osc-MDT0001@111.222.173.20@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [19104.392563] Lustre: server umount lustre-MDT0001 complete [19127.328173] LNet: 1 local NIs in recovery (showing 1): 192.168.202.128@tcp [19137.215359] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [19149.359259] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [19197.163994] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 06:15:32 (1788171332) [19219.036315] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [19222.191324] Key type lgssc unregistered [19222.551937] LNet: 602548:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [19222.563576] LNetError: 602548:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [19326.053700] LNet: Removed LNI 192.168.202.128@tcp [19327.129953] Key type .llcrypt unregistered [19327.137277] Key type ._llcrypt unregistered [19347.586434] Key type ._llcrypt registered [19347.587979] Key type .llcrypt registered [19347.713620] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [19349.258317] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [19349.311951] alg: No test for adler32 (adler32-zlib) [19350.475855] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [19350.731655] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [19352.471775] Key type lgssc registered [19353.790801] Lustre: Echo OBD driver; http://www.lustre.org/ [19359.150428] LNet: Added LNI 192.168.202.128@tcp100 [8/256/0/180] [19360.289774] LNet: Added LNI 192.168.202.128@tcp101 [8/256/0/180] [19361.375595] LNet: Added LNI 192.168.202.128@tcp102 [8/256/0/180] [19362.273989] LNet: Added LNI 192.168.202.128@tcp103 [8/256/0/180] [19363.302831] LNet: Added LNI 192.168.202.128@tcp104 [8/256/0/180] [19364.596356] LNet: Added LNI 192.168.202.128@tcp105 [8/256/0/180] [19365.863083] LNet: Added LNI 192.168.202.128@tcp106 [8/256/0/180] [19367.148377] LNet: Added LNI 192.168.202.128@tcp107 [8/256/0/180] [19368.431921] LNet: Added LNI 192.168.202.128@tcp108 [8/256/0/180] [19369.904114] LNet: Added LNI 192.168.202.128@tcp109 [8/256/0/180] [19371.238319] LNet: Added LNI 192.168.202.128@tcp110 [8/256/0/180] [19372.442637] LNet: Added LNI 192.168.202.128@tcp111 [8/256/0/180] [19373.629975] LNet: Added LNI 192.168.202.128@tcp112 [8/256/0/180] [19375.264578] LNet: Added LNI 192.168.202.128@tcp113 [8/256/0/180] [19377.231813] LNet: Added LNI 192.168.202.128@tcp114 [8/256/0/180] [19378.537568] LNet: Added LNI 192.168.202.128@tcp115 [8/256/0/180] [19379.746172] LNet: Added LNI 192.168.202.128@tcp116 [8/256/0/180] [19381.014369] LNet: Added LNI 192.168.202.128@tcp117 [8/256/0/180] [19382.396065] LNet: Added LNI 192.168.202.128@tcp118 [8/256/0/180] [19383.647972] LNet: Added LNI 192.168.202.128@tcp119 [8/256/0/180] [19384.946278] LNet: Added LNI 192.168.202.128@tcp120 [8/256/0/180] [19386.547892] LNet: Added LNI 192.168.202.128@tcp121 [8/256/0/180] [19387.966965] LNet: Added LNI 192.168.202.128@tcp122 [8/256/0/180] [19389.453589] LNet: Added LNI 192.168.202.128@tcp123 [8/256/0/180] [19390.679435] LNet: Added LNI 192.168.202.128@tcp124 [8/256/0/180] [19392.143727] LNet: Added LNI 192.168.202.128@tcp125 [8/256/0/180] [19393.377455] LNet: Added LNI 192.168.202.128@tcp126 [8/256/0/180] [19394.339902] LNet: Added LNI 192.168.202.128@tcp127 [8/256/0/180] [19395.293661] LNet: Added LNI 192.168.202.128@tcp128 [8/256/0/180] [19396.405710] LNet: Added LNI 192.168.202.128@tcp129 [8/256/0/180] [19397.049458] LNet: Added LNI 192.168.202.128@tcp130 [8/256/0/180] [19397.621441] LNet: Added LNI 192.168.202.128@tcp131 [8/256/0/180] [19398.667193] LNet: Added LNI 192.168.202.128@tcp132 [8/256/0/180] [19399.674986] LNet: Added LNI 192.168.202.128@tcp133 [8/256/0/180] [19401.001665] LNet: Added LNI 192.168.202.128@tcp134 [8/256/0/180] [19402.315372] LNet: Added LNI 192.168.202.128@tcp135 [8/256/0/180] [19410.284411] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [19410.398407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19411.685436] Lustre: 605585:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.128@tcp: can't add NID 192.168.202.128@tcp130: rc = -75 [19411.923134] LNetError: 603042:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.128 [19411.966602] LNetError: Protocol error connecting to 192.168.202.128@tcp117 at host 192.168.202.128:988: is it running a compatible version of Lustre? [19411.991696] LNetError: Skipped 3 previous similar messages [19412.143621] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [19412.194285] LNet: 7 local NIs in recovery (showing 7): 192.168.202.128@tcp125, 192.168.202.128@tcp126, 192.168.202.128@tcp127, 192.168.202.128@tcp128, 192.168.202.128@tcp124, 192.168.202.128@tcp130, 192.168.202.128@tcp129 [19412.200266] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [19412.374192] Lustre: lustre-MDT0000: new disk, initializing [19412.533078] LNetError: 605617:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.128 [19412.545546] LNetError: 605617:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 20 previous similar messages [19412.551518] Lustre: 605642:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.128@tcp: can't add NID 192.168.202.128@tcp130: rc = -75 [19412.556539] LNetError: Protocol error connecting to 192.168.202.128@tcp131 at host 192.168.202.128:988: is it running a compatible version of Lustre? [19412.556549] LNetError: Skipped 15 previous similar messages [19412.571841] Lustre: 605642:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [19412.607543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [19412.673359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [19414.243685] LNetError: 605605:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.128 [19414.261160] LNetError: 605605:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 7 previous similar messages [19414.283983] LNetError: Protocol error connecting to 192.168.202.128@tcp126 at host 192.168.202.128:988: is it running a compatible version of Lustre? [19414.311106] LNetError: Skipped 8 previous similar messages [19418.345543] LNetError: 603041:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.128 [19418.366437] LNetError: 603041:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 6 previous similar messages [19418.387507] LNetError: Protocol error connecting to 192.168.202.128@tcp126 at host 192.168.202.128:988: is it running a compatible version of Lustre? [19418.397138] LNetError: Skipped 5 previous similar messages [19419.508658] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [19426.530358] LNetError: 603043:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.128 [19426.535197] LNetError: 603043:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 4 previous similar messages [19426.541217] LNetError: Protocol error connecting to 192.168.202.128@tcp126 at host 192.168.202.128:988: is it running a compatible version of Lustre? [19426.562479] LNetError: Skipped 4 previous similar messages [19430.341380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19430.498849] Lustre: 605601:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [19430.584626] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [19430.596947] Lustre: Skipped 1 previous similar message [19430.803527] Lustre: lustre-MDT0001: new disk, initializing [19430.855046] Lustre: 606405:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.128@tcp: can't add NID 192.168.202.128@tcp130: rc = -75 [19430.867372] Lustre: 606405:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [19430.989819] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [19431.045710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [19431.062544] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [19434.879543] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [19436.554966] Lustre: 606746:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.128@tcp: can't add NID 192.168.202.128@tcp130: rc = -75 [19436.566646] Lustre: 606746:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [19437.800625] LNetError: 605614:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.128 [19437.825520] LNetError: 605614:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 13 previous similar messages [19437.831699] LNetError: Protocol error connecting to 192.168.202.128@tcp133 at host 192.168.202.128:988: is it running a compatible version of Lustre? [19437.842089] LNetError: Skipped 13 previous similar messages [19445.535165] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [19451.444566] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [19454.195497] LNetError: 605609:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.128 [19454.219154] LNetError: 605609:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 17 previous similar messages [19454.240306] LNetError: Protocol error connecting to 192.168.202.128@tcp128 at host 192.168.202.128:988: is it running a compatible version of Lustre? [19454.255298] LNetError: Skipped 17 previous similar messages [19458.885596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [19459.175413] Lustre: lustre-OST0000: new disk, initializing [19459.178529] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [19459.185325] Lustre: 607575:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [19459.293930] Lustre: 607580:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.128@tcp: can't add NID 192.168.202.128@tcp130: rc = -75 [19459.308231] Lustre: 607580:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [19459.376808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [19466.276969] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [19467.344228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [19467.361764] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [19467.426065] Lustre: 607993:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.128@tcp: can't add NID 192.168.202.128@tcp130: rc = -75 [19467.448770] Lustre: 607993:0:(lustre_peer.c:124:class_add_uuid()) Skipped 17 previous similar messages [19467.595869] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [19472.611198] LNet: 6 local NIs in recovery (showing 6): 192.168.202.128@tcp133, 192.168.202.128@tcp124, 192.168.202.128@tcp130, 192.168.202.128@tcp129, 192.168.202.128@tcp132, 192.168.202.128@tcp126 [19477.136264] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [19486.181877] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19486.191493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [19486.215466] Lustre: Skipped 1 previous similar message [19486.241648] Lustre: Skipped 1 previous similar message [19488.993303] LNetError: 603042:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.128 [19489.023187] LNetError: 603042:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 32 previous similar messages [19489.051331] LNetError: Protocol error connecting to 192.168.202.128@tcp122 at host 192.168.202.128:988: is it running a compatible version of Lustre? [19489.059895] LNetError: Skipped 32 previous similar messages [19491.305079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [19491.319183] Lustre: Skipped 1 previous similar message [19492.069660] Lustre: server umount lustre-OST0000 complete [19496.426046] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19496.436709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [19496.446908] Lustre: Skipped 1 previous similar message [19496.459824] Lustre: Skipped 1 previous similar message [19501.544736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [19501.556651] Lustre: Skipped 1 previous similar message [19502.481638] Lustre: server umount lustre-MDT0000 complete [19506.815031] LustreError: 605594:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788171644 with bad export cookie 4807541025532425967 [19506.825038] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [19507.221954] Lustre: server umount lustre-MDT0001 complete [19519.597685] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [19522.934984] Key type lgssc unregistered [19523.467593] LNet: 609494:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [19523.481352] LNetError: 609494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [19524.511253] LNet: Removed LNI 192.168.202.128@tcp [19524.514316] LNetError: 609494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [19524.523188] LNet: Removed LNI 192.168.202.128@tcp100 [19525.599233] LNet: Removed LNI 192.168.202.128@tcp101 [19525.606199] LNetError: 609494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [19525.620179] LNetError: 609494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [19525.634315] LNet: Removed LNI 192.168.202.128@tcp102 [19526.687197] LNet: Removed LNI 192.168.202.128@tcp103 [19527.711167] LNet: Removed LNI 192.168.202.128@tcp104 [19527.712939] LNetError: 609494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [19527.745100] LNetError: 609494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [19527.773836] LNet: Removed LNI 192.168.202.128@tcp105 [19528.799270] LNet: Removed LNI 192.168.202.128@tcp106 [19529.825447] LNet: Removed LNI 192.168.202.128@tcp107 [19530.847466] LNet: Removed LNI 192.168.202.128@tcp108 [19531.871660] LNet: Removed LNI 192.168.202.128@tcp109 [19531.873530] LNetError: 609494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [19531.878849] LNetError: 609494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [19531.892465] LNet: Removed LNI 192.168.202.128@tcp110 [19532.895223] LNet: Removed LNI 192.168.202.128@tcp111 [19533.919400] LNet: Removed LNI 192.168.202.128@tcp112 [19534.943251] LNet: Removed LNI 192.168.202.128@tcp113 [19535.975350] LNet: Removed LNI 192.168.202.128@tcp114 [19536.992333] LNet: Removed LNI 192.168.202.128@tcp115 [19538.016019] LNet: Removed LNI 192.168.202.128@tcp116 [19539.039305] LNet: Removed LNI 192.168.202.128@tcp117 [19540.127212] LNet: Removed LNI 192.168.202.128@tcp118 [19540.135954] LNetError: 609494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [19540.162881] LNetError: 609494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [19540.179239] LNet: Removed LNI 192.168.202.128@tcp119 [19541.216034] LNet: Removed LNI 192.168.202.128@tcp120 [19542.243346] LNet: Removed LNI 192.168.202.128@tcp121 [19543.263132] LNet: Removed LNI 192.168.202.128@tcp122 [19544.287196] LNet: Removed LNI 192.168.202.128@tcp123 [19544.296992] LNet: Removed LNI 192.168.202.128@tcp124 [19545.311243] LNet: Removed LNI 192.168.202.128@tcp125 [19545.321230] LNet: Removed LNI 192.168.202.128@tcp126 [19546.341818] LNet: Removed LNI 192.168.202.128@tcp127 [19547.359280] LNet: Removed LNI 192.168.202.128@tcp128 [19547.363879] LNet: Removed LNI 192.168.202.128@tcp129 [19547.367350] LNet: Removed LNI 192.168.202.128@tcp130 [19548.383146] LNet: Removed LNI 192.168.202.128@tcp131 [19548.387958] LNet: Removed LNI 192.168.202.128@tcp132 [19548.391877] LNet: Removed LNI 192.168.202.128@tcp133 [19549.407302] LNet: Removed LNI 192.168.202.128@tcp134 [19550.439853] LNet: Removed LNI 192.168.202.128@tcp135 [19551.425820] Key type .llcrypt unregistered [19551.427583] Key type ._llcrypt unregistered [19576.970401] Key type ._llcrypt registered [19576.971775] Key type .llcrypt registered [19577.144859] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [19592.394865] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [19593.846431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [19594.129190] alg: No test for adler32 (adler32-zlib) [19595.514650] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [19595.884857] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [19597.671298] Key type lgssc registered [19599.397322] Lustre: Echo OBD driver; http://www.lustre.org/ [19638.237526] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 06:22:54 (1788171774) [19658.197505] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [19661.445840] Key type lgssc unregistered [19661.840924] LNet: 613583:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [19661.848386] LNetError: 613583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [19661.858828] LNet: Removed LNI 192.168.202.128@tcp [19662.738801] Key type .llcrypt unregistered [19662.740765] Key type ._llcrypt unregistered [19680.686636] Key type ._llcrypt registered [19680.689635] Key type .llcrypt registered [19680.772278] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [19681.913735] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [19681.981850] alg: No test for adler32 (adler32-zlib) [19683.179534] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [19683.425123] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [19685.089701] Key type lgssc registered [19686.330985] Lustre: Echo OBD driver; http://www.lustre.org/ [19696.365164] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [19696.400776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19697.636976] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [19697.656355] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [19697.705451] Lustre: lustre-MDT0000: new disk, initializing [19697.805972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [19697.820646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [19702.215610] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [19713.987622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19714.119715] Lustre: 614797:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [19714.176550] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [19714.184329] Lustre: Skipped 1 previous similar message [19714.287444] Lustre: lustre-MDT0001: new disk, initializing [19714.411776] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [19714.470208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [19714.479510] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [19719.304389] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [19729.392988] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [19735.946362] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [19743.525081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [19743.825552] Lustre: lustre-OST0000: new disk, initializing [19743.831234] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [19743.839083] Lustre: 616768:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [19743.934991] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [19749.953253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [19749.964539] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [19750.113124] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [19750.852969] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [19760.744425] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [19765.484991] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing lnet_if_list [19771.381838] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [19771.425663] LNet: Added LNI 192.168.202.128@tcp10 [8/256/0/180] [19778.709625] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [19778.771015] LNet: Added LNI 192.168.202.128@tcp11 [8/256/0/180] [19785.862960] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [19785.934322] LNet: Added LNI 192.168.202.128@tcp12 [8/256/0/180] [19793.885537] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [19793.931988] LNet: Added LNI 192.168.202.128@tcp13 [8/256/0/180] [19799.901717] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [19799.923631] LNet: Added LNI 192.168.202.128@tcp14 [8/256/0/180] [19805.583042] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [19805.644602] LNet: Added LNI 192.168.202.128@tcp15 [8/256/0/180] [19810.755188] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [19810.793747] LNet: Added LNI 192.168.202.128@tcp16 [8/256/0/180] [19816.424487] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [19816.467496] LNet: Added LNI 192.168.202.128@tcp17 [8/256/0/180] [19822.403744] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [19822.446534] LNet: Added LNI 192.168.202.128@tcp18 [8/256/0/180] [19827.157857] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [19827.192477] LNet: Added LNI 192.168.202.128@tcp19 [8/256/0/180] [19832.444510] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [19832.487154] LNet: Added LNI 192.168.202.128@tcp20 [8/256/0/180] [19839.148694] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [19839.201574] LNet: Added LNI 192.168.202.128@tcp21 [8/256/0/180] [19845.905881] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [19845.944248] LNet: Added LNI 192.168.202.128@tcp22 [8/256/0/180] [19851.925670] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [19851.963245] LNet: Added LNI 192.168.202.128@tcp23 [8/256/0/180] [19858.093467] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [19858.124268] LNet: Added LNI 192.168.202.128@tcp24 [8/256/0/180] [19864.793344] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [19864.840611] LNet: Added LNI 192.168.202.128@tcp25 [8/256/0/180] [19871.726278] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [19871.749567] LNet: Added LNI 192.168.202.128@tcp26 [8/256/0/180] [19878.442973] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [19878.470210] LNet: Added LNI 192.168.202.128@tcp27 [8/256/0/180] [19885.294590] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [19885.326765] LNet: Added LNI 192.168.202.128@tcp28 [8/256/0/180] [19892.606938] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [19892.630930] LNet: Added LNI 192.168.202.128@tcp29 [8/256/0/180] [19899.327624] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [19899.369684] LNet: Added LNI 192.168.202.128@tcp30 [8/256/0/180] [19905.829905] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [19905.851507] LNet: Added LNI 192.168.202.128@tcp31 [8/256/0/180] [19911.526302] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [19911.545912] LNet: Added LNI 192.168.202.128@tcp32 [8/256/0/180] [19918.092387] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [19918.124152] LNet: Added LNI 192.168.202.128@tcp33 [8/256/0/180] [19925.947430] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [19925.975966] LNet: Added LNI 192.168.202.128@tcp34 [8/256/0/180] [19933.569509] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [19933.605658] LNet: Added LNI 192.168.202.128@tcp35 [8/256/0/180] [19942.115249] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [19942.162806] LNet: Added LNI 192.168.202.128@tcp36 [8/256/0/180] [19950.713368] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [19950.742110] LNet: Added LNI 192.168.202.128@tcp37 [8/256/0/180] [19958.870668] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [19958.894454] LNet: Added LNI 192.168.202.128@tcp38 [8/256/0/180] [19966.333081] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [19966.369613] LNet: Added LNI 192.168.202.128@tcp39 [8/256/0/180] [19974.144494] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [19974.164327] LNet: Added LNI 192.168.202.128@tcp40 [8/256/0/180] [19982.621485] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [19982.693471] LNet: Added LNI 192.168.202.128@tcp41 [8/256/0/180] [19991.369642] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [19991.393840] LNet: Added LNI 192.168.202.128@tcp42 [8/256/0/180] [19998.597627] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [19998.628488] LNet: Added LNI 192.168.202.128@tcp43 [8/256/0/180] [20006.175674] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [20006.211775] LNet: Added LNI 192.168.202.128@tcp44 [8/256/0/180] [20014.551857] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [20014.588778] LNet: Added LNI 192.168.202.128@tcp45 [8/256/0/180] [20024.400149] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [20024.434671] LNet: Added LNI 192.168.202.128@tcp46 [8/256/0/180] [20036.102285] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [20036.130632] LNet: Added LNI 192.168.202.128@tcp47 [8/256/0/180] [20047.792383] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [20047.827295] LNet: Added LNI 192.168.202.128@tcp48 [8/256/0/180] [20056.669733] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [20056.707508] LNet: Added LNI 192.168.202.128@tcp49 [8/256/0/180] [20065.586693] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [20065.633671] LNet: Added LNI 192.168.202.128@tcp50 [8/256/0/180] [20083.174876] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20083.184038] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [20083.186622] Lustre: Skipped 1 previous similar message [20083.187687] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [20088.298792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [20088.314700] Lustre: Skipped 4 previous similar messages [20088.486433] Lustre: server umount lustre-MDT0000 complete [20093.036885] LustreError: 614791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788172230 with bad export cookie 8357642529137282440 [20093.040446] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [20093.047227] LustreError: 614791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [20093.416746] Lustre: server umount lustre-MDT0001 complete [20109.791488] Lustre: 614071:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788172231/real 1788172231] req@0000000065c49ee6 x1875034055555328/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788172247 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [20109.838386] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20109.854695] Lustre: Skipped 1 previous similar message [20114.688845] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [20114.911529] Lustre: 614069:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788172236/real 1788172236] req@00000000ae1265b0 x1875034055555456/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788172252 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [20119.009108] Lustre: 614069:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788172240/real 1788172240] req@0000000036c51dc2 x1875034055555712/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788172256 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [20124.127644] Lustre: 614071:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788172245/real 1788172245] req@0000000007e3ab95 x1875034055555840/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788172261 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [20124.733131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [20137.503700] LustreError: 614067:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000043c8e47f x1875034055556608/t0(0) o250->MGC192.168.202.128@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [20137.890887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [20138.915327] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [20142.103901] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [20145.129263] LustreError: 616761:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [20145.147413] LustreError: 616761:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [20150.252758] LustreError: 616762:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [20152.428670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [20152.723631] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [20154.810567] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [20159.074281] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [20168.273583] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [20174.263620] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [20198.891659] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [20198.898872] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20198.918744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [20200.940543] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [20200.947454] Lustre: Skipped 2 previous similar messages [20203.844285] Lustre: server umount lustre-OST0000 complete [20209.126470] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [20209.131789] LustreError: Skipped 1 previous similar message [20209.144780] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20209.169421] Lustre: Skipped 1 previous similar message [20209.173697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [20214.057790] Lustre: server umount lustre-MDT0000 complete [20216.295088] LustreError: 642983:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [20216.320974] LustreError: 642983:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [20219.254098] LustreError: 614792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788172357 with bad export cookie 8357642529137285135 [20219.270286] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [20219.276188] LustreError: 614792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [20219.710630] Lustre: server umount lustre-MDT0001 complete [20231.743466] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [20235.600830] Key type lgssc unregistered [20236.137878] LNet: 643816:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [20236.152747] LNetError: 643816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [20236.180821] LNet: Removed LNI 192.168.202.128@tcp [20236.184364] LNet: Removed LNI 192.168.202.128@tcp10 [20236.196238] LNet: Removed LNI 192.168.202.128@tcp11 [20236.200522] LNet: Removed LNI 192.168.202.128@tcp12 [20236.207032] LNet: Removed LNI 192.168.202.128@tcp13 [20236.215889] LNet: Removed LNI 192.168.202.128@tcp14 [20236.222972] LNet: Removed LNI 192.168.202.128@tcp15 [20236.230741] LNet: Removed LNI 192.168.202.128@tcp16 [20236.242059] LNet: Removed LNI 192.168.202.128@tcp17 [20236.246706] LNet: Removed LNI 192.168.202.128@tcp18 [20236.253967] LNet: Removed LNI 192.168.202.128@tcp19 [20236.262751] LNet: Removed LNI 192.168.202.128@tcp20 [20236.275934] LNet: Removed LNI 192.168.202.128@tcp21 [20236.287755] LNet: Removed LNI 192.168.202.128@tcp22 [20236.301508] LNet: Removed LNI 192.168.202.128@tcp23 [20236.307296] LNet: Removed LNI 192.168.202.128@tcp24 [20236.325573] LNet: Removed LNI 192.168.202.128@tcp25 [20236.339532] LNet: Removed LNI 192.168.202.128@tcp26 [20236.350869] LNet: Removed LNI 192.168.202.128@tcp27 [20236.361719] LNet: Removed LNI 192.168.202.128@tcp28 [20236.366885] LNet: Removed LNI 192.168.202.128@tcp29 [20236.368770] LNet: Removed LNI 192.168.202.128@tcp30 [20236.378577] LNet: Removed LNI 192.168.202.128@tcp31 [20236.382299] LNet: Removed LNI 192.168.202.128@tcp32 [20236.385107] LNet: Removed LNI 192.168.202.128@tcp33 [20236.388711] LNet: Removed LNI 192.168.202.128@tcp34 [20236.391793] LNet: Removed LNI 192.168.202.128@tcp35 [20236.394899] LNet: Removed LNI 192.168.202.128@tcp36 [20236.398131] LNet: Removed LNI 192.168.202.128@tcp37 [20236.401665] LNet: Removed LNI 192.168.202.128@tcp38 [20236.405366] LNet: Removed LNI 192.168.202.128@tcp39 [20236.409643] LNet: Removed LNI 192.168.202.128@tcp40 [20236.425300] LNet: Removed LNI 192.168.202.128@tcp41 [20236.434754] LNet: Removed LNI 192.168.202.128@tcp42 [20236.438840] LNet: Removed LNI 192.168.202.128@tcp43 [20236.445599] LNet: Removed LNI 192.168.202.128@tcp44 [20236.451735] LNet: Removed LNI 192.168.202.128@tcp45 [20236.459897] LNet: Removed LNI 192.168.202.128@tcp46 [20236.476111] LNet: Removed LNI 192.168.202.128@tcp47 [20236.478204] LNet: Removed LNI 192.168.202.128@tcp48 [20236.487164] LNet: Removed LNI 192.168.202.128@tcp49 [20236.514993] LNet: Removed LNI 192.168.202.128@tcp50 [20237.684130] Key type .llcrypt unregistered [20237.687281] Key type ._llcrypt unregistered [20264.008299] Key type ._llcrypt registered [20264.009757] Key type .llcrypt registered [20264.137770] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [20281.813556] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [20283.273083] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [20283.514790] alg: No test for adler32 (adler32-zlib) [20284.786985] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [20285.090098] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [20286.863203] Key type lgssc registered [20288.372576] Lustre: Echo OBD driver; http://www.lustre.org/ [20333.038121] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [20333.065277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [20334.379550] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [20334.446238] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [20334.521489] Lustre: lustre-MDT0000: new disk, initializing [20334.618067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [20334.640146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [20339.910354] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [20352.436958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [20352.621630] Lustre: 647015:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [20352.713735] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [20352.717818] Lustre: Skipped 1 previous similar message [20352.796859] Lustre: lustre-MDT0001: new disk, initializing [20352.879208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [20352.914287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [20352.937042] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [20358.270692] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [20371.398417] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [20380.002494] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [20389.988882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [20390.406763] Lustre: *** cfs_fail_loc=90f, val=0*** [20390.492615] Lustre: lustre-OST0000: new disk, initializing [20390.498311] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [20390.503295] Lustre: 649034:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [20390.615192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [20396.686434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [20396.713484] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [20396.895618] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [20399.112454] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [20410.760689] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [20422.627792] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20422.630182] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [20422.657453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [20422.658680] Lustre: Skipped 1 previous similar message [20425.765487] Lustre: server umount lustre-OST0000 complete [20429.792750] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [20429.803925] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20429.830751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [20429.842831] Lustre: Skipped 1 previous similar message [20432.873335] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20432.875659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [20432.905182] Lustre: Skipped 1 previous similar message [20435.751471] Lustre: server umount lustre-MDT0000 complete [20440.479979] LustreError: 649033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788172578 with bad export cookie 13614461519263802127 [20440.482499] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [20440.498286] LustreError: 649033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [20441.130115] Lustre: server umount lustre-MDT0001 complete [20453.897079] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [20457.430986] Key type lgssc unregistered [20457.760836] LNet: 650941:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [20457.767974] LNetError: 650941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [20457.786951] LNet: Removed LNI 192.168.202.128@tcp [20458.849509] Key type .llcrypt unregistered [20458.851617] Key type ._llcrypt unregistered [20481.620395] Key type ._llcrypt registered [20481.624407] Key type .llcrypt registered [20481.740783] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [20482.765984] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [20482.834266] alg: No test for adler32 (adler32-zlib) [20484.104537] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [20484.253844] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [20485.895563] Key type lgssc registered [20486.761637] Lustre: Echo OBD driver; http://www.lustre.org/ [20498.108701] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [20498.164612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [20499.819536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [20505.670766] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [20509.158090] LustreError: 652214:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [20509.196783] LustreError: 652214:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [20514.275532] LustreError: 652213:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [20517.051086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [20517.501350] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [20523.305602] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [20531.963923] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [20537.586130] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [20545.816727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [20546.549240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [20555.448699] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [20564.120955] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [20580.873381] loop0: detected capacity change from 0 to 204800000 [20581.042419] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [20581.050904] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [20581.061660] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [20587.436436] loop0: detected capacity change from 0 to 204800000 [20587.567599] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [20587.596029] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [20587.635203] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [20593.690538] loop0: detected capacity change from 0 to 204800000 [20600.172401] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [20600.351699] Lustre: fs15246-OST0000: new disk, initializing [20600.358202] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [20600.367509] Lustre: 655825:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [20600.447398] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [20608.257730] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [20619.248900] loop1: detected capacity change from 0 to 204800000 [20625.642554] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [20625.747384] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [20625.828658] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [20626.002853] Lustre: fs15246-MDT0000: new disk, initializing [20626.268994] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [20626.313486] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [20626.487146] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [20627.552326] Lustre: 656947:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000008355273b x1875034895338624/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788172781 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [20631.580766] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [20632.694900] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [20632.714447] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [20632.770308] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [20658.149531] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20658.167303] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [20659.968435] Lustre: server umount fs15246-MDT0000 complete [20663.276638] LustreError: 652208:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [20668.404825] LustreError: 653526:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [20671.902228] Lustre: server umount fs15246-OST0000 complete [20685.792457] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [20685.805766] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20685.830540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [20687.841889] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [20687.849115] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20687.884376] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [20690.097390] Lustre: server umount lustre-OST0000 complete [20696.554872] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [20696.576063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20696.608674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [20696.610754] Lustre: Skipped 2 previous similar messages [20704.225024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [20704.232125] Lustre: Skipped 3 previous similar messages [20709.346567] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [20709.585755] Lustre: server umount lustre-MDT0000 complete [20713.465668] LustreError: 653526:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [20713.505208] LustreError: 653526:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [20715.001952] LustreError: 654057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788172852 with bad export cookie 17139591579163231489 [20715.012134] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [20715.894781] Lustre: server umount lustre-MDT0001 complete [20729.594378] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [20732.664832] Key type lgssc unregistered [20733.049655] LNet: 660047:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [20733.054697] LNetError: 660047:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [20733.078259] LNet: Removed LNI 192.168.202.128@tcp [20733.965488] Key type .llcrypt unregistered [20733.971994] Key type ._llcrypt unregistered [20754.819094] Key type ._llcrypt registered [20754.820466] Key type .llcrypt registered [20754.980933] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [20756.336100] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [20756.508046] alg: No test for adler32 (adler32-zlib) [20757.891649] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [20758.124643] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [20759.807440] Key type lgssc registered [20761.051811] Lustre: Echo OBD driver; http://www.lustre.org/ [20808.729658] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [20819.669761] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [20862.699806] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [20862.745174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [20864.071223] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [20864.094812] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [20864.141897] Lustre: lustre-MDT0000: new disk, initializing [20864.238471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [20864.274592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [20870.126115] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [20884.053779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [20884.177877] Lustre: 664517:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [20884.258568] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [20884.263283] Lustre: Skipped 1 previous similar message [20884.373551] Lustre: lustre-MDT0001: new disk, initializing [20884.549644] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [20884.601155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [20884.609525] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [20889.805784] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [20899.975923] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [20906.528500] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [20913.914694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [20914.191660] Lustre: lustre-OST0000: new disk, initializing [20914.194380] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [20914.199546] Lustre: 666488:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [20914.257377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [20920.727447] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [20924.532364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [20924.546786] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [20924.698866] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [20931.448885] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [20937.383069] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [20937.788561] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [20943.874780] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [20944.097456] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [20950.507605] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [20950.511951] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20950.518152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [20950.526370] Lustre: Skipped 1 previous similar message [20950.535908] Lustre: Skipped 1 previous similar message [20952.714377] Lustre: server umount lustre-OST0000 complete [20960.742300] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20960.753557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [20960.759801] Lustre: Skipped 1 previous similar message [20960.774887] Lustre: Skipped 1 previous similar message [20962.994846] Lustre: server umount lustre-MDT0000 complete [20967.346309] LustreError: 664510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788173105 with bad export cookie 17223795128864575273 [20967.353893] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [20967.827284] Lustre: server umount lustre-MDT0001 complete [20994.930620] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [21004.983981] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [21038.955838] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 20744 sec ========= 06:46:14 (1788173174) [21040.827962] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 06:46:16 (1788173176) === [21044.509151] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 06:46:20 (1788173180) === [21087.911460] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [21091.325702] Key type lgssc unregistered [21091.664708] LNet: 673135:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [21091.668996] LNetError: 673135:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [21091.680248] LNet: Removed LNI 192.168.202.128@tcp [21092.552681] Key type .llcrypt unregistered [21092.554773] Key type ._llcrypt unregistered