[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 488773636 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003304] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.007902] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009009] pid_max: default: 32768 minimum: 301 [ 0.010117] LSM: Security Framework initializing [ 0.011034] Yama: becoming mindful. [ 0.012054] SELinux: Initializing. [ 0.013072] *** VALIDATE selinux *** [ 0.022184] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026763] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027181] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028090] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029112] *** VALIDATE tmpfs *** [ 0.030492] *** VALIDATE proc *** [ 0.031281] *** VALIDATE cgroup *** [ 0.032007] *** VALIDATE cgroup2 *** [ 0.033206] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035034] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037020] Spectre V2 : User space: Vulnerable [ 0.038005] Speculative Store Bypass: Vulnerable [ 0.040788] debug: unmapping init [mem 0xffffffffb7259000-0xffffffffb7260fff] [ 0.042191] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043611] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044028] ... version: 2 [ 0.045014] ... bit width: 48 [ 0.046011] ... generic registers: 4 [ 0.047014] ... value mask: 0000ffffffffffff [ 0.048018] ... max period: 00007fffffffffff [ 0.049026] ... fixed-purpose events: 3 [ 0.050014] ... event mask: 000000070000000f [ 0.051267] rcu: Hierarchical SRCU implementation. [ 0.053531] smp: Bringing up secondary CPUs ... [ 0.054665] x86: Booting SMP configuration: [ 0.055029] .... node #0, CPUs: #1 #2 #3 [ 0.058393] smp: Brought up 1 node, 4 CPUs [ 0.060013] smpboot: Max logical packages: 1 [ 0.061028] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.146181] node 0 deferred pages initialised in 83ms [ 0.149139] devtmpfs: initialized [ 0.150187] x86/mm: Memory block size: 128MB [ 0.152412] gcov: version magic: 0x41383552 [ 0.153549] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.155064] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.156233] pinctrl core: initialized pinctrl subsystem [ 0.157196] [ 0.157521] ************************************************************* [ 0.158008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.159009] ** ** [ 0.160010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.161011] ** ** [ 0.162011] ** This means that this kernel is built to expose internal ** [ 0.163013] ** IOMMU data structures, which may compromise security on ** [ 0.164014] ** your system. ** [ 0.165014] ** ** [ 0.166008] ** If you see this message and you are not debugging the ** [ 0.167007] ** kernel, report this immediately to your vendor! ** [ 0.168010] ** ** [ 0.169008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.170010] ************************************************************* [ 0.171578] NET: Registered protocol family 16 [ 0.172342] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.173045] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.174057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.175431] cpuidle: using governor menu [ 0.177701] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.180529] PCI: Using configuration type 1 for base access [ 0.182127] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.191090] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.192035] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.193166] cryptd: max_cpu_qlen set to 1000 [ 0.195204] ACPI: Added _OSI(Module Device) [ 0.196013] ACPI: Added _OSI(Processor Device) [ 0.197015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.198011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.202063] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.204607] ACPI: Interpreter enabled [ 0.205068] ACPI: PM: (supports S0 S3 S4 S5) [ 0.206017] ACPI: Using IOAPIC for interrupt routing [ 0.207134] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.208433] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.218544] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.219044] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.220017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.221085] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.223063] acpiphp: Slot [2] registered [ 0.224028] acpiphp: Slot [5] registered [ 0.224937] acpiphp: Slot [6] registered [ 0.225083] acpiphp: Slot [7] registered [ 0.226060] acpiphp: Slot [8] registered [ 0.227075] acpiphp: Slot [9] registered [ 0.228094] acpiphp: Slot [10] registered [ 0.229048] acpiphp: Slot [3] registered [ 0.230147] acpiphp: Slot [4] registered [ 0.231103] acpiphp: Slot [11] registered [ 0.232103] acpiphp: Slot [12] registered [ 0.233097] acpiphp: Slot [13] registered [ 0.234089] acpiphp: Slot [14] registered [ 0.235090] acpiphp: Slot [15] registered [ 0.236108] acpiphp: Slot [16] registered [ 0.237080] acpiphp: Slot [17] registered [ 0.238095] acpiphp: Slot [18] registered [ 0.239051] acpiphp: Slot [19] registered [ 0.240091] acpiphp: Slot [20] registered [ 0.241070] acpiphp: Slot [21] registered [ 0.242070] acpiphp: Slot [22] registered [ 0.243110] acpiphp: Slot [23] registered [ 0.244151] acpiphp: Slot [24] registered [ 0.245059] acpiphp: Slot [25] registered [ 0.246079] acpiphp: Slot [26] registered [ 0.247162] acpiphp: Slot [27] registered [ 0.247910] acpiphp: Slot [28] registered [ 0.248144] acpiphp: Slot [29] registered [ 0.249138] acpiphp: Slot [30] registered [ 0.250100] acpiphp: Slot [31] registered [ 0.251083] PCI host bridge to bus 0000:00 [ 0.252019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.253022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.254019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.255024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.256021] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.257033] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.258200] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.259906] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.261355] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.267016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.270417] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.271021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.272022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.273021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.274609] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.277865] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.280121] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.283033] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.288020] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.300016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.306017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.311816] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.323024] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.332019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.348017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.360456] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.371021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.379018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.401025] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.410088] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.421018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.431018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.455019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.468000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.475016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.481020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.499021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.509524] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.515021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.521017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.537015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.545174] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.552017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.560019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.600056] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.612028] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.614335] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.618406] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.620393] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.622215] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.626156] iommu: Default domain type: Passthrough [ 0.629405] SCSI subsystem initialized [ 0.631148] ACPI: bus type USB registered [ 0.632087] usbcore: registered new interface driver usbfs [ 0.634071] usbcore: registered new interface driver hub [ 0.635062] usbcore: registered new device driver usb [ 0.637185] pps_core: LinuxPPS API ver. 1 registered [ 0.639010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.641047] PTP clock support registered [ 0.643074] EDAC MC: Ver: 3.0.0 [ 0.645153] PCI: Using ACPI for IRQ routing [ 0.646661] NetLabel: Initializing [ 0.648015] NetLabel: domain hash size = 128 [ 0.649010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.651083] NetLabel: unlabeled traffic allowed by default [ 0.654095] vgaarb: loaded [ 0.656033] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.657006] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.661311] clocksource: Switched to clocksource kvm-clock [ 0.771266] VFS: Disk quotas dquot_6.6.0 [ 0.772280] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.773826] *** VALIDATE ramfs *** [ 0.774493] *** VALIDATE hugetlbfs *** [ 0.775399] pnp: PnP ACPI init [ 0.776959] pnp: PnP ACPI: found 6 devices [ 0.799691] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.801533] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.802796] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.804107] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.805807] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.807145] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.808803] NET: Registered protocol family 2 [ 0.810347] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.813700] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.816871] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.820568] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.822842] TCP: Hash tables configured (established 65536 bind 65536) [ 0.825187] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.827401] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.829349] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.832182] NET: Registered protocol family 1 [ 0.834831] RPC: Registered named UNIX socket transport module. [ 0.836119] RPC: Registered udp transport module. [ 0.837055] RPC: Registered tcp transport module. [ 0.838041] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.839482] NET: Registered protocol family 44 [ 0.841202] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.843427] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.846067] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.848863] PCI: CLS 0 bytes, default 64 [ 0.851135] Unpacking initramfs... [ 2.274877] debug: unmapping init [mem 0xffff92707cc54000-0xffff92707ffbffff] [ 2.279057] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.281553] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.284745] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.802163] Initialise system trusted keyrings [ 2.803982] Key type blacklist registered [ 2.806095] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.816710] zbud: loaded [ 2.821500] *** VALIDATE nfs *** [ 2.822917] *** VALIDATE nfs4 *** [ 2.824714] pstore: using deflate compression [ 2.828223] Platform Keyring initialized [ 2.945812] NET: Registered protocol family 38 [ 2.947851] Key type asymmetric registered [ 2.949541] Asymmetric key parser 'x509' registered [ 2.951563] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.954963] io scheduler mq-deadline registered [ 2.957159] io scheduler kyber registered [ 2.959235] io scheduler bfq registered [ 2.961374] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.964863] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.968514] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.974228] ACPI: Power Button [PWRF] [ 2.980743] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.989521] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.005302] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.011048] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.023750] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.051919] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.080169] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.085314] Non-volatile memory driver v1.3 [ 3.086587] Linux agpgart interface v0.103 [ 3.122349] virtio_blk virtio1: [vda] 149760 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.125596] vda: detected capacity change from 0 to 76677120 [ 3.143522] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.146429] vdb: detected capacity change from 0 to 1073741824 [ 3.162079] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.165313] vdc: detected capacity change from 0 to 2621440000 [ 3.180093] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.182244] vdd: detected capacity change from 0 to 2621440000 [ 3.195569] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.197932] vde: detected capacity change from 0 to 4294967296 [ 3.217654] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.220455] vdf: detected capacity change from 0 to 4294967296 [ 3.229268] libphy: Fixed MDIO Bus: probed [ 3.238188] usbcore: registered new interface driver usbserial_generic [ 3.240374] usbserial: USB Serial support registered for generic [ 3.242544] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.246508] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.248750] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.251618] mousedev: PS/2 mouse device common for all mice [ 3.256361] rtc_cmos 00:05: RTC can wake from S4 [ 3.259837] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.264161] rtc_cmos 00:05: registered as rtc0 [ 3.266241] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.269403] intel_pstate: CPU model not supported [ 3.272416] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.274388] hid: raw HID events driver (C) Jiri Kosina [ 3.278144] usbcore: registered new interface driver usbhid [ 3.280238] usbhid: USB HID core driver [ 3.281959] drop_monitor: Initializing network drop monitor service [ 3.282349] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.284526] Initializing XFRM netlink socket [ 3.290157] NET: Registered protocol family 10 [ 3.293426] Segment Routing with IPv6 [ 3.294989] NET: Registered protocol family 17 [ 3.296959] mpls_gso: MPLS GSO support [ 3.304744] RAS: Correctable Errors collector initialized. [ 3.306831] AVX version of gcm_enc/dec engaged. [ 3.307828] AES CTR mode by8 optimization enabled [ 3.379862] sched_clock: Marking stable (3379841708, 0)->(4396519058, -1016677350) [ 3.382670] registered taskstats version 1 [ 3.383924] Loading compiled-in X.509 certificates [ 3.385251] zswap: loaded using pool lzo/zbud [ 3.411271] Key type big_key registered [ 3.423603] Key type encrypted registered [ 3.425133] ima: No TPM chip found, activating TPM-bypass! [ 3.426527] ima: Allocated hash algorithm: sha1 [ 3.427986] ima: No architecture policies found [ 3.429721] evm: Initialising EVM extended attributes: [ 3.431611] evm: security.selinux [ 3.432808] evm: security.ima [ 3.434037] evm: security.capability [ 3.435424] evm: HMAC attrs: 0x1 [ 3.437942] rtc_cmos 00:05: setting system clock to 2026-09-03 12:20:59 UTC (1788438059) [ 3.443547] debug: unmapping init [mem 0xffffffffb8203000-0xffffffffb83fffff] [ 3.445760] debug: unmapping init [mem 0xffffffffb6f82000-0xffffffffb7258fff] [ 3.455143] Write protecting the kernel read-only data: 28672k [ 3.457908] debug: unmapping init [mem 0xffffffffb5603000-0xffffffffb57fffff] [ 3.460946] debug: unmapping init [mem 0xffffffffb5f14000-0xffffffffb5ffffff] [ 3.495218] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.502820] systemd[1]: Detected virtualization kvm. [ 3.504489] systemd[1]: Detected architecture x86-64. [ 3.506424] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.531333] systemd[1]: No hostname configured. [ 3.533050] systemd[1]: Set hostname to . [ 3.535100] random: systemd: uninitialized urandom read (16 bytes read) [ 3.537299] systemd[1]: Initializing machine ID from random generator. [ 3.589240] random: ln: uninitialized urandom read (6 bytes read) [ 3.676943] random: systemd: uninitialized urandom read (16 bytes read) [ 3.680294] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.689160] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ 3.700766] systemd[1]: Starting Apply Kernel Variables... Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local File Systems. Starting Setup Virtual Console... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Sockets. [ OK ] Reached target Paths. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Swap. [ OK ] Reached target Slices. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.254764] device-mapper: uevent: version 1.0.3 [ 4.256927] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.038639] random: fast init done [ 5.083691] virtio_net virtio0 ens2: renamed from eth0 [ 5.184207] scsi host0: ata_piix [ 5.202134] scsi host1: ata_piix [ 5.203439] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.205869] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.901345] dracut-initqueue[582]: RTNETLINK answers: File exists [ 9.940319] random: crng init done [ 9.941876] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.408944] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.617783] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.912837] SELinux: Disabled at runtime. [ 11.974295] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.982619] systemd[1]: Detected virtualization kvm. [ 11.984494] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.525466] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.530949] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.537885] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.541204] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.544182] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.549941] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.563328] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Initrd File Systems. Mounting Huge Pages File System... [ 12.617306] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Paths. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Slices. Starting Apply Kernel Variables... Mounting Kernel Debug File System... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.037919] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.319053] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.348360] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.489357] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.500684] EDAC sbridge: Ver: 1.1.2 [ 15.168374] Key type dns_resolver registered [ 15.474068] NFS: Registering the id_resolver key type [ 15.476128] Key type id_resolver registered [ 15.477988] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg236-server login: [ 59.861906] libcfs: loading out-of-tree module taints kernel. [ 59.902753] Key type ._llcrypt registered [ 59.905766] Key type .llcrypt registered [ 60.013552] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 82.150458] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 84.093518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 84.109162] alg: No test for adler32 (adler32-zlib) [ 85.746320] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 86.727329] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 88.650673] Key type lgssc registered [ 90.601808] Lustre: Echo OBD driver; http://www.lustre.org/ [ 110.828819] hrtimer: interrupt took 12458013 ns [ 112.122731] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 153.794373] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 167.472920] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 167.504896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 168.824404] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 168.871839] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 168.955362] Lustre: lustre-MDT0000: new disk, initializing [ 169.015700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 169.030222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 173.572763] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 188.655646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 188.796751] Lustre: 6506:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 188.831390] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 188.839485] Lustre: Skipped 1 previous similar message [ 188.947674] Lustre: lustre-MDT0001: new disk, initializing [ 188.998975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 189.018168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 189.022921] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 195.085789] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 200.875213] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 213.414179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 213.705360] Lustre: lustre-OST0000: new disk, initializing [ 213.711436] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 213.721896] Lustre: 8446:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 213.782290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 215.592553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 215.600968] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 215.691307] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 221.360218] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 240.000105] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 240.248328] Lustre: lustre-OST0001: new disk, initializing [ 240.255475] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 240.261474] Lustre: 9520:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 240.417787] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 246.854957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 246.870836] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 246.952186] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 247.512202] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 260.408063] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 272.625527] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 279.548801] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing check_logdir /tmp/testlogs/ [ 286.847237] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing yml_node [ 292.016590] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 294.803899] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 297.637236] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 299.559276] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 08:25:54 EDT 2026 [ 317.342435] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 318.772785] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 337.380254] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 337.394696] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 337.411317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 338.923395] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 338.924385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 338.951307] Lustre: Skipped 2 previous similar messages [ 338.960888] Lustre: Skipped 2 previous similar messages [ 340.845499] Lustre: server umount lustre-MDT0000 complete [ 345.148897] LustreError: 6499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438401 with bad export cookie 13343837865343328716 [ 345.157331] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 345.166139] LustreError: 6499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 349.154387] LustreError: 6516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 349.155598] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 349.156487] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 349.156492] Lustre: Skipped 1 previous similar message [ 349.173865] LustreError: 6516:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 349.190339] Lustre: Skipped 1 previous similar message [ 350.179218] LustreError: 6516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 350.193561] LustreError: 6516:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 351.703940] Lustre: server umount lustre-MDT0001 complete [ 355.746762] Lustre: server umount lustre-OST0000 complete [ 361.585044] Lustre: server umount lustre-OST0001 complete [ 369.284702] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 381.514745] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 421.959544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 422.321854] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 422.361950] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 422.453532] Lustre: lustre-MDT0000: new disk, initializing [ 422.533791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 422.557103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 427.749370] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 439.339049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 439.522858] Lustre: 15082:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 439.598242] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 439.600824] Lustre: Skipped 1 previous similar message [ 439.651265] Lustre: lustre-MDT0001: new disk, initializing [ 439.772942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 439.825878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 439.838167] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 445.685761] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 456.263831] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 463.505696] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 470.993631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 471.262911] Lustre: lustre-OST0000: new disk, initializing [ 471.266825] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 471.272450] Lustre: 17052:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 471.337357] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 473.395424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 473.405366] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 473.547364] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 480.024361] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 492.860626] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 499.591815] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 500.045670] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 507.725208] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 508.205576] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 514.532922] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 514.534588] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 514.567811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 514.575035] LustreError: Skipped 1 previous similar message [ 514.595430] Lustre: Skipped 4 previous similar messages [ 517.621323] Lustre: server umount lustre-OST0000 complete [ 524.770653] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 524.776220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 524.783714] Lustre: Skipped 2 previous similar messages [ 524.808200] Lustre: Skipped 1 previous similar message [ 528.526415] Lustre: server umount lustre-MDT0000 complete [ 529.897374] LustreError: 15094:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 529.918316] LustreError: 15094:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 533.113565] LustreError: 18593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438589 with bad export cookie 13343837865343330249 [ 533.115904] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 533.128653] LustreError: 18593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 533.638779] Lustre: server umount lustre-MDT0001 complete [ 538.342847] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 08:29:53 (1788438593) [ 540.955655] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 543.020431] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 08:29:57 (1788438597) [ 550.088191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 550.687583] LustreError: 19463:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 550.896338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 556.003639] LustreError: 19464:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 556.053350] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 565.229622] LustreError: 19464:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 565.271793] LustreError: 19464:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 566.099154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 571.165520] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 578.827706] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 586.235644] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 593.354240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 602.060514] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 612.149546] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 626.660603] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 626.667748] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 626.701444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 629.004582] Lustre: server umount lustre-OST0000 complete [ 639.775814] Lustre: server umount lustre-MDT0000 complete [ 642.017644] LustreError: 19460:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 642.034713] LustreError: 19460:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 643.810615] LustreError: 19445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438699 with bad export cookie 13343837865343331131 [ 643.822468] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 643.834494] LustreError: 19445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 644.197133] Lustre: server umount lustre-MDT0001 complete [ 657.833625] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 660.677566] Key type lgssc unregistered [ 660.986846] LNet: 22995:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 661.000458] LNetError: 22995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 661.030611] LNet: Removed LNI 192.168.202.136@tcp [ 661.891713] Key type .llcrypt unregistered [ 661.895973] Key type ._llcrypt unregistered [ 685.690876] Key type ._llcrypt registered [ 685.694112] Key type .llcrypt registered [ 685.777372] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 686.590763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 686.621511] alg: No test for adler32 (adler32-zlib) [ 687.740336] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 687.948619] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 689.600677] Key type lgssc registered [ 690.667698] Lustre: Echo OBD driver; http://www.lustre.org/ [ 698.854262] loop: module loaded [ 698.957396] loop0: detected capacity change from 0 to 204800000 [ 699.179980] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 699.205905] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 699.419931] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 705.779401] loop0: detected capacity change from 0 to 204800000 [ 706.013312] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 706.040306] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 706.054205] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 711.586904] loop0: detected capacity change from 0 to 204800000 [ 717.649611] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 717.677760] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 719.051638] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 719.173882] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 719.296452] Lustre: lustre-MDT0000: new disk, initializing [ 719.456194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 719.504542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 725.259447] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 737.091444] loop1: detected capacity change from 0 to 204800000 [ 745.397433] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 745.962900] Lustre: lustre-OST07c6: new disk, initializing [ 745.970773] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 745.976266] Lustre: Skipped 1 previous similar message [ 745.983196] Lustre: 26183:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 746.092161] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 752.331826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 752.346469] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 752.414135] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 755.722877] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 762.179051] Lustre: 25169:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 762.187693] Lustre: 25169:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 762.190946] Lustre: 25169:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 762.198906] Lustre: 25169:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 762.205498] Lustre: 25169:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 762.211303] Lustre: 25169:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 780.428300] Lustre: 25169:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 780.438672] Lustre: 25169:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 780.446587] Lustre: 25169:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 780.460669] Lustre: 25169:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 780.474820] Lustre: 25169:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 780.488717] Lustre: 25169:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 785.916892] Lustre: 26849:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 785.928539] Lustre: 26849:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 785.939321] Lustre: 26849:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 785.950914] Lustre: 26849:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 785.958238] Lustre: 26849:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 785.970910] Lustre: 26849:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 785.979309] Lustre: 26849:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 785.986520] Lustre: 26849:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 785.997822] Lustre: 26849:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 786.003209] Lustre: 26849:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 786.015828] Lustre: 26849:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 786.025048] Lustre: 26849:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 791.616413] Lustre: 25168:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 791.636361] Lustre: 25168:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 791.642860] Lustre: 25168:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 791.664430] Lustre: 25168:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 791.678408] Lustre: 25168:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 791.692167] Lustre: 25168:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 791.703750] Lustre: 25168:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 791.716586] Lustre: 25168:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 791.731791] Lustre: 25168:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 791.742669] Lustre: 25168:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 791.765499] Lustre: 25168:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 791.777647] Lustre: 25168:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 800.007626] Lustre: 25168:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 800.030491] Lustre: 25168:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 800.043194] Lustre: 25168:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 800.053145] Lustre: 25168:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 800.060994] Lustre: 25168:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 800.088654] Lustre: 25168:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 800.110728] Lustre: 25168:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 800.124119] Lustre: 25168:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 800.144974] Lustre: 25168:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 800.157524] Lustre: 25168:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 800.177049] Lustre: 25168:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 800.188452] Lustre: 25168:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 808.744573] Lustre: 25170:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 808.765836] Lustre: 25170:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 808.777641] Lustre: 25170:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 808.792594] Lustre: 25170:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 808.802465] Lustre: 25170:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 808.811548] Lustre: 25170:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 808.822531] Lustre: 25170:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 808.833843] Lustre: 25170:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 808.841665] Lustre: 25170:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 808.849770] Lustre: 25170:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 808.855906] Lustre: 25170:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 808.862706] Lustre: 25170:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 865.249660] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 865.262931] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 865.279540] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 867.257472] Lustre: server umount lustre-OST07c6 complete [ 878.574363] Lustre: server umount lustre-MDT0000 complete [ 898.487338] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 901.626923] Key type lgssc unregistered [ 901.937570] LNet: 28740:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 901.942861] LNetError: 28740:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 901.966295] LNet: Removed LNI 192.168.202.136@tcp [ 902.920563] Key type .llcrypt unregistered [ 902.928062] Key type ._llcrypt unregistered [ 924.495947] Key type ._llcrypt registered [ 924.500716] Key type .llcrypt registered [ 924.707685] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 926.273346] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 926.444072] alg: No test for adler32 (adler32-zlib) [ 927.629994] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 927.916902] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 929.760279] Key type lgssc registered [ 931.192788] Lustre: Echo OBD driver; http://www.lustre.org/ [ 941.074707] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 941.099140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 942.575752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 947.084160] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 952.806394] LustreError: 30010:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 952.822333] LustreError: 30010:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 956.333937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 956.687386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 962.599567] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 972.106392] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 979.264696] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 986.674345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 987.197101] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 988.224811] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 994.307900] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1003.564803] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1021.922591] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1021.937618] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1021.948509] Lustre: Skipped 1 previous similar message [ 1021.977211] Lustre: Skipped 1 previous similar message [ 1026.438273] Lustre: server umount lustre-OST0000 complete [ 1032.164863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1032.184272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1032.189421] Lustre: Skipped 1 previous similar message [ 1037.281664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1037.284370] Lustre: Skipped 1 previous similar message [ 1037.732849] Lustre: server umount lustre-MDT0000 complete [ 1042.391304] LustreError: 29990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439098 with bad export cookie 11036365192254297172 [ 1042.393642] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1042.405163] LustreError: 29990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1043.120145] Lustre: server umount lustre-MDT0001 complete [ 1057.950739] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 1062.045351] Key type lgssc unregistered [ 1062.414927] LNet: 33749:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1062.421377] LNetError: 33749:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1062.458922] LNet: Removed LNI 192.168.202.136@tcp [ 1063.367840] Key type .llcrypt unregistered [ 1063.370492] Key type ._llcrypt unregistered [ 1085.107501] Key type ._llcrypt registered [ 1085.110116] Key type .llcrypt registered [ 1085.251129] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 1086.418567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1086.496962] alg: No test for adler32 (adler32-zlib) [ 1087.984604] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1088.308412] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 1090.048233] Key type lgssc registered [ 1091.054921] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1099.047246] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1099.081740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1100.605557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1105.499624] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1109.988766] LustreError: 35015:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1110.004359] LustreError: 35015:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1114.505915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1114.931623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1120.283343] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1128.215025] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1134.838465] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1142.967673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1143.421991] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1145.549423] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1149.870488] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1158.764647] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1174.659346] Lustre: server umount lustre-OST0000 complete [ 1180.642235] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1180.651310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1180.656736] Lustre: Skipped 1 previous similar message [ 1180.671970] Lustre: Skipped 1 previous similar message [ 1185.770503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1185.773243] Lustre: Skipped 1 previous similar message [ 1190.885647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1190.894698] Lustre: Skipped 1 previous similar message [ 1190.976383] Lustre: server umount lustre-MDT0000 complete [ 1191.843628] Lustre: 34327:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788439231/real 1788439231] req@000000009aafd610 x1875314363888128/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788439247 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1191.879233] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1191.908591] LustreError: 35015:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1191.938215] LustreError: 35015:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1195.185731] LustreError: 36859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439251 with bad export cookie 10204096282384044143 [ 1195.196848] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1195.208992] LustreError: 36859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1195.715618] Lustre: server umount lustre-MDT0001 complete [ 1210.198316] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 1214.020923] Key type lgssc unregistered [ 1214.383644] LNet: 38569:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1214.393335] LNetError: 38569:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1214.409893] LNet: Removed LNI 192.168.202.136@tcp [ 1215.650498] Key type .llcrypt unregistered [ 1215.665186] Key type ._llcrypt unregistered [ 1243.403731] Key type ._llcrypt registered [ 1243.408477] Key type .llcrypt registered [ 1243.507778] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 1244.339863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1244.456491] alg: No test for adler32 (adler32-zlib) [ 1245.677644] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1246.000523] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 1247.712253] Key type lgssc registered [ 1248.722366] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1257.558421] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1257.608995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1259.350333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1264.377746] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1269.800346] LustreError: 39983:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1269.832600] LustreError: 39983:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1274.850867] LustreError: 39984:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1278.075456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1278.612516] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1285.860703] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1295.865270] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1303.597846] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1311.900958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1312.292772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1320.172538] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1329.677529] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1330.766460] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1339.879781] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1339.893798] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1339.911938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1340.764662] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.36@tcp (stopping) [ 1340.898591] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1340.919199] Lustre: Skipped 1 previous similar message [ 1342.150250] Lustre: server umount lustre-MDT0000 complete [ 1345.835064] LustreError: 39979:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1346.414900] LustreError: 39965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439402 with bad export cookie 18348088805676023738 [ 1346.443674] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1346.448609] LustreError: 39965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1350.968139] LustreError: 39980:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1350.995543] LustreError: 39980:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1351.007832] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.36@tcp (stopping) [ 1351.030993] Lustre: Skipped 3 previous similar messages [ 1351.143749] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1353.277384] Lustre: server umount lustre-MDT0001 complete [ 1382.884567] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1383.430915] Lustre: server umount lustre-OST0000 complete [ 1406.338828] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 1410.012676] Key type lgssc unregistered [ 1410.561877] LNet: 43850:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1410.570613] LNetError: 43850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1410.585473] LNet: Removed LNI 192.168.202.136@tcp [ 1412.385440] Key type .llcrypt unregistered [ 1412.388172] Key type ._llcrypt unregistered [ 1441.495461] Key type ._llcrypt registered [ 1441.498240] Key type .llcrypt registered [ 1441.601457] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 1442.708269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1443.173217] alg: No test for adler32 (adler32-zlib) [ 1444.256874] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1444.542313] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 1446.208193] Key type lgssc registered [ 1447.695769] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1456.971844] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1457.002598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1458.588289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1463.873517] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1468.898706] LustreError: 45264:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1468.919643] LustreError: 45264:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1474.020662] LustreError: 45265:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1474.344315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1474.755659] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1480.477478] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1488.506265] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1495.838202] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1503.738813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1504.197524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1511.528278] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1519.705971] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1528.825632] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1535.818386] Lustre: lustre-OST0000: Not available for connect from 192.168.202.36@tcp (stopping) [ 1539.050442] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1539.056864] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1539.070858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1539.090050] Lustre: Skipped 1 previous similar message [ 1540.918879] Lustre: lustre-OST0000: Not available for connect from 192.168.202.36@tcp (stopping) [ 1540.926621] Lustre: Skipped 1 previous similar message [ 1541.671333] Lustre: server umount lustre-OST0000 complete [ 1562.083386] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1562.096227] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1562.119240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1564.644687] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1569.761441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1569.769217] Lustre: Skipped 2 previous similar messages [ 1575.393141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1577.673661] Lustre: server umount lustre-MDT0000 complete [ 1578.979468] LustreError: 45260:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1579.009589] LustreError: 45260:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1582.205795] LustreError: 45999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439638 with bad export cookie 14676654969680454793 [ 1582.218587] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1582.226570] LustreError: 45999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1588.723911] Lustre: server umount lustre-MDT0001 complete [ 1604.457925] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 1608.121334] Key type lgssc unregistered [ 1608.571929] LNet: 48997:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1608.581460] LNetError: 48997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1608.613825] LNet: Removed LNI 192.168.202.136@tcp [ 1609.829831] Key type .llcrypt unregistered [ 1609.836143] Key type ._llcrypt unregistered [ 1643.678099] Key type ._llcrypt registered [ 1643.679808] Key type .llcrypt registered [ 1643.777669] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 1645.024954] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1645.037483] alg: No test for adler32 (adler32-zlib) [ 1646.062936] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1646.288004] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 1648.008162] Key type lgssc registered [ 1649.721735] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1660.431047] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1660.469267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1661.995394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1667.026575] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1672.166193] LustreError: 50419:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1672.201489] LustreError: 50419:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1677.295203] LustreError: 50420:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1677.878338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1678.304051] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1683.576890] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1692.481897] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1699.742968] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1707.776153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1707.945194] LustreError: 52241:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1708.123891] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1713.172993] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1715.699939] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1725.981990] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1731.263802] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1735.228399] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1748.327390] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-03 8h50m01s [ 1750.680489] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1752.912537] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.36@tcp (stopping) [ 1753.960831] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.36@tcp (stopping) [ 1754.085299] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1754.109541] Lustre: Skipped 1 previous similar message [ 1755.195030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1755.205901] Lustre: Skipped 5 previous similar messages [ 1758.862577] Lustre: server umount lustre-MDT0000 complete [ 1759.047955] LustreError: 52254:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1762.841451] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1765.176393] LustreError: 50415:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1765.196652] LustreError: 50415:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1770.484067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1770.691159] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1770.939592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1771.013338] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1771.951631] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1776.254972] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1780.044960] Lustre: DEBUG MARKER: Wait for df (36723) ... [ 1782.189589] Lustre: DEBUG MARKER: done [ 1789.922542] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1789.937214] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1789.953310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1791.457572] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1791.462486] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1793.780524] Lustre: server umount lustre-OST0000 complete [ 1802.724986] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1802.727710] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1802.753254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1802.763232] Lustre: Skipped 4 previous similar messages [ 1804.090543] Lustre: server umount lustre-MDT0000 complete [ 1808.197335] LustreError: 50446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439864 with bad export cookie 15356861252816044988 [ 1808.206071] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1808.211615] LustreError: 50446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1808.504963] Lustre: server umount lustre-MDT0001 complete [ 1821.257048] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 1824.282856] Key type lgssc unregistered [ 1824.638839] LNet: 55343:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1824.650318] LNetError: 55343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1825.705475] LNet: Removed LNI 192.168.202.136@tcp [ 1826.637171] Key type .llcrypt unregistered [ 1826.639471] Key type ._llcrypt unregistered [ 1871.597352] Key type ._llcrypt registered [ 1871.599791] Key type .llcrypt registered [ 1871.743274] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 1872.977584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1873.028254] alg: No test for adler32 (adler32-zlib) [ 1874.030494] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1874.304028] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 1876.024256] Key type lgssc registered [ 1876.963825] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1886.876167] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1886.893712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1888.224696] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1888.255674] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1888.557457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1894.985444] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1906.191707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1906.260320] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1906.283092] Lustre: 57549:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1906.602340] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1911.388494] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1919.284465] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1926.399968] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1935.603681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1935.850056] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1936.206748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1944.718220] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 1944.933603] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1955.664968] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1964.752362] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-03 8h53m39s [ 1966.442760] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1973.771527] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1976.589910] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1976.731674] LustreError: 57554:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1997.140411] Lustre: lustre-MDT0000: Client 06c83690-74d2-4b3d-9be9-4ca3bf4f74f9 (at 192.168.202.36@tcp) reconnecting [ 2016.753403] LustreError: 57554:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 2019.545515] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 2023.557683] Lustre: DEBUG MARKER: done [ 2032.112199] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2032.120488] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2032.121322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2032.137952] Lustre: Skipped 1 previous similar message [ 2032.157936] LustreError: Skipped 1 previous similar message [ 2036.643781] Lustre: server umount lustre-OST0000 complete [ 2042.341140] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2042.342598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2042.368220] Lustre: Skipped 1 previous similar message [ 2042.404443] Lustre: Skipped 2 previous similar messages [ 2043.429684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2047.478257] Lustre: server umount lustre-MDT0000 complete [ 2052.691573] LustreError: 57540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440108 with bad export cookie 7398368743488078366 [ 2052.703046] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2052.718174] LustreError: 57540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2053.150563] Lustre: server umount lustre-MDT0001 complete [ 2070.388416] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 2075.191970] Key type lgssc unregistered [ 2075.646329] LNet: 61880:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2075.659469] LNetError: 61880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2075.677637] LNet: Removed LNI 192.168.202.136@tcp [ 2076.918142] Key type .llcrypt unregistered [ 2076.921245] Key type ._llcrypt unregistered [ 2126.451462] Key type ._llcrypt registered [ 2126.455682] Key type .llcrypt registered [ 2126.580161] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 2128.115224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2128.155366] alg: No test for adler32 (adler32-zlib) [ 2129.292525] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2129.614624] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 2131.256180] Key type lgssc registered [ 2132.572492] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2135.995813] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2136.025660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2137.460454] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2137.506764] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2137.705994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2138.707677] Lustre: Failing over lustre-MDT0000 [ 2139.035955] Lustre: server umount lustre-MDT0000 complete [ 2147.353044] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 08:56:42 (1788440202) [ 2155.847418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2156.326408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2162.178245] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2171.063351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2171.197543] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2171.236953] Lustre: 64680:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2171.566243] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2176.886255] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2186.840768] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2195.677720] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2205.293118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2205.671450] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2205.979785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2214.658308] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2214.716969] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2225.022151] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2231.641495] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2243.557073] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2243.560464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2243.572095] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2243.594810] Lustre: Skipped 2 previous similar messages [ 2243.613093] Lustre: Skipped 1 previous similar message [ 2244.779174] Lustre: server umount lustre-MDT0000 complete [ 2249.254463] LustreError: 64674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440305 with bad export cookie 8339627761356048884 [ 2249.266795] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2249.275117] LustreError: 64674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2249.649244] Lustre: server umount lustre-MDT0001 complete [ 2253.440906] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2258.409491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2267.252304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2269.152162] Lustre: 63601:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788440309/real 1788440309] req@00000000fd6e7842 x1875315455472640/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788440325 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2269.181587] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2270.176165] Lustre: 63602:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788440310/real 1788440310] req@00000000d520e6b1 x1875315455472768/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788440326 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2274.336258] Lustre: 63602:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788440314/real 1788440314] req@000000001f20b917 x1875315455472896/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788440330 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2275.617702] LustreError: 66532:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2275.634924] LustreError: 66532:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2275.695045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2275.774152] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2276.705347] Lustre: 63600:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788440316/real 1788440316] req@000000007671e9da x1875315455473152/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788440332 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2278.694256] LustreError: 66533:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2279.715265] LustreError: 68452:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2279.754421] LustreError: 68452:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2281.749451] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2281.760435] Lustre: 63600:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788440321/real 1788440321] req@00000000cf6dc80a x1875315455473280/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788440337 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2285.026879] LustreError: 68452:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2285.059032] LustreError: 68452:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2290.147093] LustreError: 68453:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2290.177238] LustreError: 68453:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2291.508743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2292.127249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2297.331974] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2298.274807] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2298.375218] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2307.730140] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2314.529810] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2323.942229] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2323.945099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2323.961254] Lustre: Skipped 1 previous similar message [ 2323.971619] Lustre: Skipped 3 previous similar messages [ 2329.057872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2329.057872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2329.070015] Lustre: Skipped 1 previous similar message [ 2329.804682] Lustre: server umount lustre-MDT0000 complete [ 2334.183373] LustreError: 68454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2334.198598] LustreError: 68454:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2334.237778] LustreError: 64675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440390 with bad export cookie 8339627761356051159 [ 2334.239797] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2334.249000] LustreError: 64675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2334.776081] Lustre: server umount lustre-MDT0001 complete [ 2341.581414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2351.065727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2355.680435] Lustre: 63602:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788440395/real 1788440395] req@00000000b676b56b x1875315455504256/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788440411 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2355.715490] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2355.730776] Lustre: Skipped 1 previous similar message [ 2360.159838] LustreError: 66533:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2360.229443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2360.288350] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2364.352251] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2366.353246] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2377.476822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2383.089915] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2383.359519] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2391.852620] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2399.053849] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2408.438412] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2408.453807] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2408.489537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2419.170588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2419.178632] Lustre: Skipped 7 previous similar messages [ 2421.216592] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2421.405347] Lustre: server umount lustre-MDT0000 complete [ 2424.291244] LustreError: 71134:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2424.330862] LustreError: 71134:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 2425.556502] LustreError: 64675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440481 with bad export cookie 8339627761356052538 [ 2425.571162] LustreError: 64675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2425.573850] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2425.976558] Lustre: server umount lustre-MDT0001 complete [ 2433.055256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2436.850212] Lustre: DEBUG MARKER: files compared the same [ 2449.581907] Lustre: server umount lustre-OST0000 complete [ 2469.038510] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 2471.810542] Key type lgssc unregistered [ 2472.061619] LNet: 74427:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2472.074196] LNetError: 74427:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2472.105943] LNet: Removed LNI 192.168.202.136@tcp [ 2473.127372] Key type .llcrypt unregistered [ 2473.129683] Key type ._llcrypt unregistered [ 2498.458584] Key type ._llcrypt registered [ 2498.462056] Key type .llcrypt registered [ 2498.650514] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 2500.241768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2500.384808] alg: No test for adler32 (adler32-zlib) [ 2501.539842] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2501.934736] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 2503.688206] Key type lgssc registered [ 2505.326843] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2514.370284] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2514.407978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2515.917952] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2520.410270] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug +malloc all [ 2525.171601] LustreError: 75841:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2525.191858] LustreError: 75841:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2530.282089] LustreError: 75840:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2530.492281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2530.958385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2535.682772] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug +malloc all [ 2544.057513] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2550.599280] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2559.065366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2559.382705] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2560.451143] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2566.737262] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug +malloc all [ 2575.335764] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2591.209237] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2591.219390] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2591.221541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2592.952120] Lustre: server umount lustre-OST0000 complete [ 2597.859254] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2597.883088] LustreError: Skipped 1 previous similar message [ 2597.896150] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2597.921884] Lustre: Skipped 1 previous similar message [ 2597.957049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2597.971230] Lustre: Skipped 1 previous similar message [ 2601.442493] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2601.444839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2601.478801] Lustre: Skipped 1 previous similar message [ 2603.991567] Lustre: server umount lustre-MDT0000 complete [ 2606.561627] LustreError: 75840:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2606.596620] LustreError: 75840:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2608.548110] LustreError: 75821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440664 with bad export cookie 12896485015339616110 [ 2608.569832] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2608.941987] Lustre: server umount lustre-MDT0001 complete [ 2624.394330] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 2627.995698] Key type lgssc unregistered [ 2628.348593] LNet: 79374:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2628.357197] LNetError: 79374:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2628.367497] LNet: Removed LNI 192.168.202.136@tcp [ 2629.084111] Key type .llcrypt unregistered [ 2629.089128] Key type ._llcrypt unregistered [ 2658.311527] Key type ._llcrypt registered [ 2658.315565] Key type .llcrypt registered [ 2658.437073] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 2659.718063] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2659.949274] alg: No test for adler32 (adler32-zlib) [ 2661.059834] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2661.321743] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 2663.040250] Key type lgssc registered [ 2665.434729] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2675.976906] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2676.010838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2691.747693] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2696.867452] LustreError: 80769:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.136@tcp: failed processing log, type 4: rc = -110 [ 2723.618743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2731.920833] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2741.089413] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2752.103335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2752.193275] Lustre: *** cfs_fail_loc=706, val=0*** [ 2752.499124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2752.600148] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2757.411183] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2757.603515] LustreError: 80794:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2757.637412] LustreError: 80794:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2758.632815] LustreError: 80795:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2758.664371] LustreError: 80795:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2763.771911] LustreError: 80794:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2763.806260] LustreError: 80794:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2767.815272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2768.287843] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2773.399765] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2781.493926] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2787.855807] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2793.446502] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2793.455185] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2793.479284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2793.956776] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2793.968728] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2793.985446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2796.756746] Lustre: server umount lustre-OST0000 complete [ 2804.197488] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2804.199111] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2804.200207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2804.219218] Lustre: Skipped 1 previous similar message [ 2807.055316] Lustre: server umount lustre-MDT0000 complete [ 2809.317108] LustreError: 83209:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2809.343785] LustreError: 83209:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2811.330684] LustreError: 80776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440867 with bad export cookie 1770031096137253872 [ 2811.338131] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2811.706452] Lustre: server umount lustre-MDT0001 complete [ 2824.147520] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 2827.210496] Key type lgssc unregistered [ 2827.495531] LNet: 84386:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2827.520400] LNetError: 84386:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2827.553735] LNet: Removed LNI 192.168.202.136@tcp [ 2828.682357] Key type .llcrypt unregistered [ 2828.685083] Key type ._llcrypt unregistered [ 2860.466404] Key type ._llcrypt registered [ 2860.468347] Key type .llcrypt registered [ 2860.586607] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 2862.371062] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2862.419693] alg: No test for adler32 (adler32-zlib) [ 2863.621943] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2864.085533] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 2865.840224] Key type lgssc registered [ 2867.979891] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2878.599137] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2878.622874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2884.607763] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2893.052918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2893.423936] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2898.412654] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2898.930334] LustreError: 86493:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2898.948622] LustreError: 86493:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2903.011238] LustreError: 86492:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2903.034292] LustreError: 86492:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2907.355817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2907.650510] LustreError: 86493:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2907.692295] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2912.752206] LustreError: 86497:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2912.778933] LustreError: 86497:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2914.441900] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2917.859043] LustreError: 87240:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2917.892453] LustreError: 87240:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2923.883485] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2923.903212] Lustre: MGS: Client 856221ce-049f-4b7f-9edb-242ab59cfc4f (at 0@lo) reconnecting [ 2923.938243] LustreError: 86507:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2924.002251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [ 2924.013270] Lustre: Skipped 2 previous similar messages [ 2924.291710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2924.353424] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2930.785693] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 2946.529367] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2946.544913] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2946.563443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2950.128678] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2950.134380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2950.147530] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2950.179302] Lustre: Skipped 1 previous similar message [ 2952.369115] Lustre: server umount lustre-OST0000 complete [ 2960.363488] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2960.375664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2960.380261] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2960.382200] Lustre: Skipped 1 previous similar message [ 2963.673786] Lustre: server umount lustre-MDT0000 complete [ 2965.474358] LustreError: 86493:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2965.495823] LustreError: 86493:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2969.076263] Lustre: server umount lustre-MDT0001 complete [ 2974.355751] Lustre: server umount MGS complete [ 2988.317925] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 09:10:42 (1788441042) [ 3020.669410] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 3032.092458] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 3082.611929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3088.358584] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3099.672249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3099.946372] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3100.064298] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3100.155304] Lustre: lustre-MDT0001: new disk, initializing [ 3100.245684] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3100.284818] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3100.294853] Lustre: 93687:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3100.308187] LustreError: 93687:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3105.781461] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3118.520921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3118.807883] Lustre: lustre-OST0000: new disk, initializing [ 3118.811307] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3118.817392] Lustre: 94452:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3118.908905] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3125.686365] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3128.888311] Lustre: 94908:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000000d4c5bca x1875316226172288/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788441200 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3136.630662] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3144.383277] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3144.402350] Lustre: MGS: Client e7bf2015-777a-461c-ae5e-16053c8822ac (at 0@lo) reconnecting [ 3144.452182] Lustre: 92952:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3144.478143] LustreError: 93673:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3144.526959] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3144.640355] Lustre: lustre-MDT0000: new disk, initializing [ 3144.903220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3144.951657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3144.994303] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3146.018520] Lustre: 95524:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000008d4205bc x1875316226179072/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788441218 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3150.395842] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3151.265373] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3155.181677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3155.191429] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3155.338241] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3181.026984] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3181.033535] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3181.049281] Lustre: Skipped 1 previous similar message [ 3181.067877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3193.853731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3193.862084] Lustre: Skipped 6 previous similar messages [ 3194.848159] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3195.069600] Lustre: server umount lustre-OST0000 complete [ 3204.065296] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3204.066758] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3204.077071] LustreError: Skipped 1 previous similar message [ 3204.100537] Lustre: Skipped 2 previous similar messages [ 3205.473888] Lustre: server umount lustre-MDT0000 complete [ 3209.187301] LustreError: 93657:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3209.198885] LustreError: 93657:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3215.977714] Lustre: server umount lustre-MDT0001 complete [ 3220.256908] Lustre: server umount MGS complete [ 3232.873133] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 09:14:47 (1788441287) [ 3251.055853] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 3253.843395] Key type lgssc unregistered [ 3254.141599] LNet: 98024:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3254.148541] LNetError: 98024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3254.163056] LNet: Removed LNI 192.168.202.136@tcp [ 3255.124054] Key type .llcrypt unregistered [ 3255.127411] Key type ._llcrypt unregistered [ 3275.686806] Key type ._llcrypt registered [ 3275.690197] Key type .llcrypt registered [ 3275.781220] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 3276.676827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3276.880068] alg: No test for adler32 (adler32-zlib) [ 3278.005794] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3278.281981] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 3280.008180] Key type lgssc registered [ 3281.180452] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3290.086216] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3290.105984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3291.525756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3297.236646] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3301.864598] LustreError: 99244:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3301.878204] LustreError: 99244:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3306.995135] LustreError: 99243:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3308.113387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3308.538441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3314.523294] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3324.712799] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3331.849892] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3339.440518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3339.888954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3346.514648] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3354.441216] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3361.410182] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3374.052279] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3374.063155] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3374.079465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3377.123231] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3377.123570] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3377.146123] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3377.158890] Lustre: Skipped 1 previous similar message [ 3379.718417] Lustre: server umount lustre-OST0000 complete [ 3385.318369] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3385.328794] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3385.343773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3387.363667] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3387.364818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3387.405248] Lustre: Skipped 1 previous similar message [ 3390.487248] Lustre: server umount lustre-MDT0000 complete [ 3392.483561] LustreError: 99237:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3392.500629] LustreError: 99237:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3394.490719] LustreError: 99223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441450 with bad export cookie 2407052241304283002 [ 3394.519346] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3395.106276] Lustre: server umount lustre-MDT0001 complete [ 3408.232354] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 3411.483685] Key type lgssc unregistered [ 3411.771227] LNet: 102799:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3411.779700] LNetError: 102799:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3411.800548] LNet: Removed LNI 192.168.202.136@tcp [ 3412.713837] Key type .llcrypt unregistered [ 3412.715870] Key type ._llcrypt unregistered [ 3431.583911] Key type ._llcrypt registered [ 3431.596064] Key type .llcrypt registered [ 3431.828188] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 3433.049070] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3433.166872] alg: No test for adler32 (adler32-zlib) [ 3434.311832] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3434.639975] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 3436.464368] Key type lgssc registered [ 3437.394515] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3443.436056] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3443.443769] LustreError: 103878:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 3443.453433] LustreError: 103879:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 3443.463445] LustreError: 103878:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4986 [ 3443.474122] LustreError: 103878:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3443.481240] LustreError: 103878:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3443.506683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3445.184945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3453.558297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3453.847452] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3458.529619] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3463.721341] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing lsmod [ 3466.719397] LustreError: 105082:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 3466.750428] LustreError: 105086:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 3466.757388] LustreError: 105082:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4967 [ 3466.763581] LustreError: 105082:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3466.779306] LustreError: 105082:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3466.796116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3467.242185] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3472.360894] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3472.372542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3472.380143] Lustre: Skipped 2 previous similar messages [ 3476.390387] Lustre: server umount lustre-MDT0000 complete [ 3477.479200] LustreError: 103917:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3477.503412] LustreError: 103917:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3478.505104] LustreError: 103917:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3478.515713] LustreError: 103917:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3480.489631] LustreError: 105124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441536 with bad export cookie 533898822136258007 [ 3480.492624] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3480.500330] LustreError: 105124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3480.608505] LustreError: 104349:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000d6cee646 x1875316823838592/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3480.876866] Lustre: server umount lustre-MDT0001 complete [ 3495.134677] Lustre: server umount lustre-OST0000 complete [ 3505.913935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3506.448211] LustreError: 106152:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3506.581302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3511.178961] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3511.781175] LustreError: 106151:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3516.902820] LustreError: 106152:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3521.798188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3522.438194] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3528.291152] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3537.019771] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3544.651097] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3552.552197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3553.001852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3559.123343] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3567.714678] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3576.564391] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3587.042527] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3587.047613] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3587.067972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3587.095078] Lustre: Skipped 3 previous similar messages [ 3587.114818] Lustre: Skipped 1 previous similar message [ 3591.417480] Lustre: server umount lustre-OST0000 complete [ 3597.281201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3597.300302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3597.308926] Lustre: Skipped 1 previous similar message [ 3597.328545] Lustre: Skipped 1 previous similar message [ 3601.613125] Lustre: server umount lustre-MDT0000 complete [ 3602.415873] LustreError: 106148:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3602.438639] LustreError: 106148:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3605.554381] LustreError: 106131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441661 with bad export cookie 533898822136258714 [ 3605.560097] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3605.565550] LustreError: 106131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3605.868495] Lustre: server umount lustre-MDT0001 complete [ 3617.601224] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 3620.788633] Key type lgssc unregistered [ 3621.128397] LNet: 109716:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3621.133280] LNetError: 109716:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3621.159248] LNet: Removed LNI 192.168.202.136@tcp [ 3622.006377] Key type .llcrypt unregistered [ 3622.009604] Key type ._llcrypt unregistered [ 3651.044400] Key type ._llcrypt registered [ 3651.048968] Key type .llcrypt registered [ 3651.238776] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 3653.101253] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3653.251106] alg: No test for adler32 (adler32-zlib) [ 3654.265877] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3654.684039] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 3656.464486] Key type lgssc registered [ 3658.011464] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3666.247358] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3666.257894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3667.825358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3672.850481] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3677.157377] LustreError: 111131:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3677.181758] LustreError: 111131:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3682.279235] LustreError: 111130:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3683.963079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3684.512912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3690.831894] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3700.042683] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3706.954408] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3715.762098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3716.126291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3722.835161] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3731.402329] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3737.650938] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3746.988128] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3751.268607] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3758.053309] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3758.055757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3758.071075] Lustre: Skipped 1 previous similar message [ 3758.087962] Lustre: Skipped 1 previous similar message [ 3763.170288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3763.181835] Lustre: Skipped 1 previous similar message [ 3768.299281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3770.336208] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3770.471451] Lustre: server umount lustre-OST0000 complete [ 3776.481730] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3776.490542] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3776.516170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3776.525954] Lustre: Skipped 1 previous similar message [ 3778.541886] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3783.657680] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3783.661935] Lustre: Skipped 3 previous similar messages [ 3789.280390] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3789.564544] Lustre: server umount lustre-MDT0000 complete [ 3793.896852] LustreError: 111131:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3793.913575] LustreError: 111131:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3794.208666] LustreError: 111112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441850 with bad export cookie 2858170737248511683 [ 3794.217860] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3794.221769] LustreError: 111112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3794.765494] Lustre: server umount lustre-MDT0001 complete [ 3807.397493] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 3810.806525] Key type lgssc unregistered [ 3811.110522] LNet: 114791:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3811.114706] LNetError: 114791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3811.122936] LNet: Removed LNI 192.168.202.136@tcp [ 3812.306852] Key type .llcrypt unregistered [ 3812.308500] Key type ._llcrypt unregistered [ 3837.231545] Key type ._llcrypt registered [ 3837.237555] Key type .llcrypt registered [ 3837.410569] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 3839.011713] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3839.082318] alg: No test for adler32 (adler32-zlib) [ 3840.296812] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3840.557686] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 3842.248364] Key type lgssc registered [ 3843.677164] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3851.984194] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3852.001360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3853.515811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3859.561532] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3863.015957] LustreError: 116010:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3863.048467] LustreError: 116010:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3868.133088] LustreError: 116009:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3869.570860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3869.845401] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3874.561459] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3883.161831] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3889.515566] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3897.803384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3898.288598] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3899.333365] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3906.547411] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 3915.884972] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3930.082457] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3930.089904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3930.104174] Lustre: Skipped 1 previous similar message [ 3930.130762] Lustre: Skipped 1 previous similar message [ 3932.090433] Lustre: server umount lustre-OST0000 complete [ 3936.739795] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3936.751098] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3936.783802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3940.330497] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3940.340278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3940.381613] Lustre: Skipped 1 previous similar message [ 3942.528711] Lustre: server umount lustre-MDT0000 complete [ 3945.443860] LustreError: 116006:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3945.468136] LustreError: 116006:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3946.616957] LustreError: 116745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442002 with bad export cookie 7221456752897183153 [ 3946.620115] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3946.630018] LustreError: 116745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3946.906607] Lustre: server umount lustre-MDT0001 complete [ 3959.489555] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 3962.494352] Key type lgssc unregistered [ 3962.865097] LNet: 119547:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3962.880554] LNetError: 119547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3962.895662] LNet: Removed LNI 192.168.202.136@tcp [ 3963.909887] Key type .llcrypt unregistered [ 3963.916924] Key type ._llcrypt unregistered [ 3988.422719] Key type ._llcrypt registered [ 3988.424578] Key type .llcrypt registered [ 3988.547114] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 3990.013059] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3990.053036] alg: No test for adler32 (adler32-zlib) [ 3991.092742] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3991.388742] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 3993.096110] Key type lgssc registered [ 3994.165631] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4002.294461] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4002.332367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4003.884760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4008.833709] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4013.027279] LustreError: 120961:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4013.057030] LustreError: 120961:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4018.146861] LustreError: 120960:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4018.686707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4019.116116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4023.837227] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4032.700642] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4039.023571] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4046.462285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4046.785831] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4048.824038] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 4053.629845] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4060.840963] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4071.182936] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 4075.182662] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 4075.190322] Lustre: Skipped 2 previous similar messages [ 4077.197580] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 4077.258268] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 4080.501416] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 4080.515031] Lustre: Skipped 2 previous similar messages [ 4084.509566] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4084.511792] Lustre: Skipped 1 previous similar message [ 4091.889604] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.36@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 4091.898210] Lustre: Skipped 2 previous similar messages [ 4094.126980] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.36@tcp 192.168.0.[2,10]@tcp [ 4099.060331] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.36/24@tcp in log lustre-MDT0000 [ 4099.066628] Lustre: Skipped 2 previous similar messages [ 4102.658156] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.36/24@tcp [ 4102.670071] Lustre: Skipped 1 previous similar message [ 4109.815105] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4109.818749] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4109.824278] LustreError: Skipped 1 previous similar message [ 4109.842624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4110.319102] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4110.327609] LustreError: Skipped 4 previous similar messages [ 4110.336380] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4110.346367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4110.350084] Lustre: Skipped 1 previous similar message [ 4115.433629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4115.776551] Lustre: server umount lustre-OST0000 complete [ 4120.550630] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4120.558199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4120.564790] Lustre: Skipped 1 previous similar message [ 4125.668643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4125.679695] Lustre: Skipped 1 previous similar message [ 4134.368135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4134.700235] Lustre: server umount lustre-MDT0000 complete [ 4135.907141] LustreError: 121690:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4135.922874] LustreError: 121690:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4138.684112] LustreError: 120942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442194 with bad export cookie 8432149978367851010 [ 4138.685485] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4138.702080] LustreError: 120942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4139.114194] Lustre: server umount lustre-MDT0001 complete [ 4152.693558] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 4155.571189] Key type lgssc unregistered [ 4156.078121] LNet: 125759:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4156.087414] LNetError: 125759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4156.103217] LNet: Removed LNI 192.168.202.136@tcp [ 4156.858319] Key type .llcrypt unregistered [ 4156.860529] Key type ._llcrypt unregistered [ 4189.811698] Key type ._llcrypt registered [ 4189.817233] Key type .llcrypt registered [ 4189.985447] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 4191.121360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4191.158152] alg: No test for adler32 (adler32-zlib) [ 4192.240664] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 4192.628134] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 4194.312440] Key type lgssc registered [ 4195.560751] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4204.637631] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4204.690527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4206.191584] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4206.232683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4206.232944] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.36/24@tcp [ 4212.382771] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4216.808582] LustreError: 127319:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4216.826393] LustreError: 127319:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4221.124498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4221.658032] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4221.708495] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.36/24@tcp [ 4221.756699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4226.191640] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4234.531585] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4241.874856] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4250.120968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4250.726561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4251.758721] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 4257.625886] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4266.080371] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4277.731096] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4277.751714] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4277.784103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4281.830508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4281.832683] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4281.861633] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4283.874713] Lustre: server umount lustre-OST0000 complete [ 4288.481252] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4288.499703] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4288.519181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4288.522068] Lustre: Skipped 1 previous similar message [ 4292.075969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4292.079628] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4297.185500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4297.192454] Lustre: Skipped 2 previous similar messages [ 4302.304577] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4302.655491] Lustre: server umount lustre-MDT0000 complete [ 4306.863213] LustreError: 129167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442362 with bad export cookie 14796444478640260489 [ 4306.867045] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4306.886192] LustreError: 129167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4307.207437] Lustre: server umount lustre-MDT0001 complete [ 4319.105218] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 4321.918294] Key type lgssc unregistered [ 4322.244940] LNet: 130906:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4322.255701] LNetError: 130906:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4322.273625] LNet: Removed LNI 192.168.202.136@tcp [ 4323.231464] Key type .llcrypt unregistered [ 4323.234782] Key type ._llcrypt unregistered [ 4352.841366] Key type ._llcrypt registered [ 4352.843677] Key type .llcrypt registered [ 4352.957036] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 4370.531747] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 4371.475656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4371.549534] alg: No test for adler32 (adler32-zlib) [ 4372.966734] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 4373.158215] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 4374.896199] Key type lgssc registered [ 4376.273692] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4418.159491] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4418.189545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4419.526564] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4419.599060] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4419.744393] Lustre: lustre-MDT0000: new disk, initializing [ 4419.851369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4419.889448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4424.784315] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4436.339716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4436.456684] Lustre: 134303:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4436.504153] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4436.508440] Lustre: Skipped 1 previous similar message [ 4436.580599] Lustre: lustre-MDT0001: new disk, initializing [ 4436.678313] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4436.726655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4436.747435] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4440.862923] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4450.518951] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4456.469910] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4464.155330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4464.559931] Lustre: lustre-OST0000: new disk, initializing [ 4464.563534] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4464.574220] Lustre: 136271:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4464.707140] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4470.840305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4470.858989] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4470.943330] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4470.981947] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4479.563996] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4483.715685] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4483.921621] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4487.567439] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4487.804882] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4491.756404] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4491.759576] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4491.761782] LustreError: Skipped 1 previous similar message [ 4491.765717] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4491.787096] Lustre: Skipped 1 previous similar message [ 4491.816191] Lustre: Skipped 1 previous similar message [ 4495.721370] Lustre: server umount lustre-OST0000 complete [ 4501.987642] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4501.991142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4502.009039] Lustre: Skipped 1 previous similar message [ 4502.024957] Lustre: Skipped 1 previous similar message [ 4505.953262] Lustre: server umount lustre-MDT0000 complete [ 4509.188726] LustreError: 134295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442565 with bad export cookie 7246137095449054648 [ 4509.192523] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4509.199054] LustreError: 134295:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4509.658538] Lustre: server umount lustre-MDT0001 complete [ 4517.883790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4518.433517] LustreError: 138487:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4518.450901] LustreError: 138487:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4518.542853] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4523.374747] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4524.011160] LustreError: 138488:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4528.099061] LustreError: 138487:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4532.754207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4533.127867] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4537.721540] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4545.502574] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4551.804971] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4558.533780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4558.823166] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 4558.907449] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4565.317886] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4573.630187] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4578.600249] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4578.833388] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4583.619049] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4583.857571] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4593.904125] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4594.088437] Lustre: lustre-OST0001: new disk, initializing [ 4594.097425] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4594.107176] Lustre: 141865:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4594.218994] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4600.629527] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4601.993282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4602.005027] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4602.169153] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4609.422922] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4609.632112] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4614.149636] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4614.336328] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4619.972747] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9ed902d23800.ost_server_uuid 50 [ 4621.860362] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9ed902d23800.ost_server_uuid in IDLE state after 0 sec [ 4633.058482] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4633.074364] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4633.087178] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4633.099975] Lustre: Skipped 1 previous similar message [ 4633.116720] Lustre: Skipped 1 previous similar message [ 4634.070771] Lustre: server umount lustre-OST0001 complete [ 4638.197646] LustreError: 140311:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4638.215970] LustreError: 140311:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4639.715482] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4639.716689] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4639.769616] LustreError: Skipped 2 previous similar messages [ 4639.782029] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4639.809694] Lustre: Skipped 1 previous similar message [ 4643.303796] LustreError: 140525:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4643.330289] LustreError: 140525:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4645.050768] Lustre: server umount lustre-OST0000 complete [ 4650.977900] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4650.982780] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4650.992434] Lustre: Skipped 1 previous similar message [ 4651.004536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4651.013214] Lustre: Skipped 2 previous similar messages [ 4665.312186] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4665.593783] Lustre: server umount lustre-MDT0000 complete [ 4668.911134] LustreError: 138488:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4668.928155] LustreError: 138488:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4669.870906] LustreError: 138469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442725 with bad export cookie 7246137095449055523 [ 4669.875841] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4669.885435] LustreError: 138469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4670.415218] Lustre: server umount lustre-MDT0001 complete [ 4691.662461] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 4694.613344] Key type lgssc unregistered [ 4694.982772] LNet: 144867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4694.988530] LNetError: 144867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4696.041212] LNet: Removed LNI 192.168.202.136@tcp [ 4696.970515] Key type .llcrypt unregistered [ 4696.973528] Key type ._llcrypt unregistered [ 4739.379694] Key type ._llcrypt registered [ 4739.382587] Key type .llcrypt registered [ 4739.524496] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 4755.133800] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 4756.640228] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4756.830292] alg: No test for adler32 (adler32-zlib) [ 4757.931331] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 4758.278742] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 4759.984200] Key type lgssc registered [ 4761.186532] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4798.092159] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4798.126363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4799.455174] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4799.505219] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4799.645259] Lustre: lustre-MDT0000: new disk, initializing [ 4799.775605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4799.792311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4804.370900] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4816.381878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4816.502406] Lustre: 149022:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4816.577846] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4816.582556] Lustre: Skipped 1 previous similar message [ 4816.716289] Lustre: lustre-MDT0001: new disk, initializing [ 4816.928441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4817.023255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4817.051514] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4821.858878] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4832.405187] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4838.445523] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4844.866646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4845.389316] Lustre: lustre-OST0000: new disk, initializing [ 4845.397882] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4845.413613] Lustre: 150989:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4845.532297] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4851.745710] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4851.816568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4851.831669] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4851.983587] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4862.888766] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4867.979275] Lustre: Failing over lustre-OST0000 [ 4868.217812] Lustre: server umount lustre-OST0000 complete [ 4886.497461] Lustre: 147245:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788442926/real 1788442926] req@0000000056fbd2e3 x1875318212167808/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788442942 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4886.545754] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4887.516382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4887.830072] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4887.864159] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4887.900234] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4888.588401] Lustre: 147242:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788442928/real 1788442928] req@00000000563ff53d x1875318212168064/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788442944 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4891.888828] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4893.920132] Lustre: 147245:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788442933/real 1788442933] req@0000000075b31de9 x1875318212169856/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788442949 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4893.950672] Lustre: 147245:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4894.861868] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4896.628824] Lustre: lustre-OST0000: Recovery over after 0:05, of 3 clients 3 recovered and 0 were evicted. [ 4896.636044] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4898.721127] Lustre: 147242:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788442938/real 1788442938] req@0000000007b33308 x1875318212171008/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788442954 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4898.758700] Lustre: 147242:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4898.906665] Lustre: Failing over lustre-MDT0000 [ 4899.133085] Lustre: server umount lustre-MDT0000 complete [ 4901.350132] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4901.363976] Lustre: Skipped 2 previous similar messages [ 4902.880238] Lustre: 147242:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788442942/real 1788442942] req@000000007bd69724 x1875318212172032/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788442958 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4906.483550] LustreError: 149030:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4906.508257] LustreError: 149030:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 4911.587331] LustreError: 149028:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4911.602512] LustreError: 149028:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4916.712125] LustreError: 149029:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4916.743575] LustreError: 149029:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4917.600182] Lustre: 147245:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788442957/real 1788442957] req@000000006cb77214 x1875318212181120/t0(0) o400->MGC192.168.202.136@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788442973 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4917.641618] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4919.069211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4922.153865] LustreError: 149028:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4922.187273] LustreError: 149028:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4926.957656] LustreError: 150110:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4926.965759] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x280958dd97c7839a [ 4926.977377] LustreError: 150110:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 4926.993190] Lustre: MGC192.168.202.136@tcp: Connection restored to 0@lo (at 0@lo) [ 4926.996132] Lustre: Skipped 1 previous similar message [ 4927.308916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4927.446952] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4928.305380] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4932.603361] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4932.629478] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4932.710289] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4933.133763] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 4942.830443] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4942.835470] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4942.854118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4942.863423] Lustre: Skipped 1 previous similar message [ 4947.282772] Lustre: server umount lustre-OST0000 complete [ 4953.064710] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4953.079290] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4953.087812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4953.122712] Lustre: Skipped 2 previous similar messages [ 4957.760295] Lustre: server umount lustre-MDT0000 complete [ 4958.177346] LustreError: 150110:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4958.206942] LustreError: 150110:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4962.388572] LustreError: 150988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443018 with bad export cookie 2884934745064965018 [ 4962.395919] LustreError: 150988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4962.395980] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4963.046811] Lustre: server umount lustre-MDT0001 complete [ 4978.495279] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 4982.105646] Key type lgssc unregistered [ 4982.589813] LNet: 154651:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4982.606713] LNetError: 154651:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4982.634884] LNet: Removed LNI 192.168.202.136@tcp [ 4984.317607] Key type .llcrypt unregistered [ 4984.323516] Key type ._llcrypt unregistered [ 5025.301268] Key type ._llcrypt registered [ 5025.303421] Key type .llcrypt registered [ 5025.431684] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 5026.935341] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5026.976293] alg: No test for adler32 (adler32-zlib) [ 5028.303802] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 5028.692573] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 5030.512277] Key type lgssc registered [ 5031.850704] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5041.093376] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5041.122726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5042.492820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5047.160923] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5052.898930] LustreError: 156172:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5052.930812] LustreError: 156172:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5056.781480] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5058.019825] LustreError: 156173:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5061.754821] LustreError: 156201:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5061.767784] LustreError: 156201:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5061.776487] LustreError: 156201:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 5062.474207] Lustre: server umount lustre-MDT0000 complete [ 5079.096431] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 5092.410149] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 5134.867452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5135.052782] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5135.077819] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5135.149093] Lustre: lustre-MDT0000: new disk, initializing [ 5135.209259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5135.228680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5141.038072] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5152.996176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5153.150248] Lustre: 159512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5153.210285] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5153.215823] Lustre: Skipped 1 previous similar message [ 5153.307345] Lustre: lustre-MDT0001: new disk, initializing [ 5153.419627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5153.462427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5153.488687] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5157.578944] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5169.073161] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5176.092395] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5183.625618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5183.994546] Lustre: lustre-OST0000: new disk, initializing [ 5184.008212] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5184.025182] Lustre: 161478:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5184.138212] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5191.246793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5191.262595] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5191.445335] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5191.561463] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5203.118522] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5208.304828] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5210.741824] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 09:47:45 (1788443265) [ 5480.940153] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5480.948043] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5480.966351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5483.489963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5483.497286] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5483.519803] Lustre: Skipped 1 previous similar message [ 5486.926651] Lustre: server umount lustre-MDT0000 complete [ 5488.611792] LustreError: 159524:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5488.625235] LustreError: 159524:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5490.330144] LustreError: 159506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443546 with bad export cookie 16021683280404138623 [ 5490.331309] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5490.343972] LustreError: 159506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5493.729740] LustreError: 159519:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5493.730100] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5493.774642] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5493.782304] Lustre: Skipped 2 previous similar messages [ 5495.777503] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5497.114683] Lustre: server umount lustre-MDT0001 complete [ 5516.261500] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5516.572729] Lustre: server umount lustre-OST0000 complete [ 5531.417418] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 5543.379325] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 5588.225085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5588.478781] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5588.515396] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5588.619672] Lustre: lustre-MDT0000: new disk, initializing [ 5588.745893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5588.761271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5593.171986] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5603.753137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5603.931558] Lustre: 166153:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5603.990050] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5603.994251] Lustre: Skipped 1 previous similar message [ 5604.100704] Lustre: lustre-MDT0001: new disk, initializing [ 5604.236442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5604.299967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5604.315166] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5608.645652] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5619.244925] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5625.823071] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5633.713826] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5634.022508] Lustre: lustre-OST0000: new disk, initializing [ 5634.028376] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5634.040306] Lustre: 168119:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5634.132176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5640.489748] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5642.333796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5642.347869] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5642.473547] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5650.239952] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5654.813227] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5655.192936] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5660.634620] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5660.935373] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5663.205545] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5663.210825] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5663.213983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5663.217704] Lustre: Skipped 1 previous similar message [ 5669.509228] Lustre: server umount lustre-OST0000 complete [ 5678.561326] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5678.561820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5678.615249] Lustre: Skipped 1 previous similar message [ 5678.661310] Lustre: Skipped 4 previous similar messages [ 5691.360239] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5691.934974] Lustre: server umount lustre-MDT0000 complete [ 5693.924213] LustreError: 166159:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5693.958753] LustreError: 166159:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5697.680223] LustreError: 166145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443753 with bad export cookie 16021683280404297453 [ 5697.712487] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5698.303217] Lustre: server umount lustre-MDT0001 complete [ 5711.338337] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 09:56:05 (1788443765) [ 5729.876659] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 5741.407821] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 5786.321415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5786.551284] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5786.570698] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5786.617304] Lustre: lustre-MDT0000: new disk, initializing [ 5786.687351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5786.700802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5792.258843] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5804.340189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5804.565620] Lustre: 172878:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5804.889307] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5811.086345] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5820.896299] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5827.727834] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5836.596634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5837.185635] Lustre: lustre-OST0000: new disk, initializing [ 5837.188980] Lustre: Skipped 1 previous similar message [ 5837.191653] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5837.197276] Lustre: Skipped 2 previous similar messages [ 5837.201797] Lustre: 174859:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5844.062266] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5847.270743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5847.282902] Lustre: Skipped 1 previous similar message [ 5847.311671] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5847.466590] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5855.895239] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5868.025377] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5868.030948] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5868.042241] LustreError: Skipped 2 previous similar messages [ 5868.065760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5868.084604] Lustre: Skipped 1 previous similar message [ 5868.124187] Lustre: Skipped 5 previous similar messages [ 5870.134324] Lustre: server umount lustre-OST0000 complete [ 5876.705539] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5883.073126] Lustre: server umount lustre-MDT0000 complete [ 5886.951941] LustreError: 174070:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5887.952432] LustreError: 172870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443944 with bad export cookie 16021683280404298335 [ 5887.974992] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5887.980886] LustreError: 172870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 5888.584914] Lustre: server umount lustre-MDT0001 complete [ 5900.143916] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 09:59:14 (1788443954) [ 5918.980079] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 5930.485899] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 5970.312112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5970.654491] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5970.682669] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5970.743448] Lustre: lustre-MDT0000: new disk, initializing [ 5970.860738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5970.865345] Lustre: Skipped 2 previous similar messages [ 5970.882299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5975.025885] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 5987.329970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5987.482521] Lustre: 179185:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5987.906107] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5992.583529] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6005.735573] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6015.387678] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6023.825839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6024.361474] Lustre: 181173:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6032.468152] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6032.624351] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6033.422802] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6046.138576] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6058.474120] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6058.487501] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6058.493806] Lustre: Skipped 2 previous similar messages [ 6058.511127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6058.525581] Lustre: Skipped 4 previous similar messages [ 6061.467627] Lustre: server umount lustre-OST0000 complete [ 6071.813357] Lustre: server umount lustre-MDT0000 complete [ 6073.830201] LustreError: 179192:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6073.867891] LustreError: 179192:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6075.761602] LustreError: 179176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444131 with bad export cookie 16021683280404299420 [ 6075.767797] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6075.775026] LustreError: 179176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6076.078218] Lustre: server umount lustre-MDT0001 complete [ 6087.449829] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 6089.880761] Key type lgssc unregistered [ 6090.160660] LNet: 183031:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6090.172089] LNetError: 183031:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6090.188498] LNet: Removed LNI 192.168.202.136@tcp [ 6091.401963] Key type .llcrypt unregistered [ 6091.405209] Key type ._llcrypt unregistered [ 6113.625363] Key type ._llcrypt registered [ 6113.630227] Key type .llcrypt registered [ 6113.774860] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 6115.013828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6115.069774] alg: No test for adler32 (adler32-zlib) [ 6116.261479] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 6116.523218] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 6118.184200] Key type lgssc registered [ 6119.665580] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6128.115926] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6128.148534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6129.777496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6134.397807] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6139.381360] LustreError: 184423:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6139.408110] LustreError: 184423:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6143.784982] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6144.344947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6149.592903] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6157.432951] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6163.440257] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6170.818152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6171.330508] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6174.414917] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6179.605626] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6190.057377] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6210.532362] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6210.533939] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6210.547469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6210.558319] Lustre: Skipped 1 previous similar message [ 6210.573461] Lustre: Skipped 1 previous similar message [ 6214.422548] Lustre: server umount lustre-OST0000 complete [ 6220.769907] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6220.781819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6220.802836] Lustre: Skipped 1 previous similar message [ 6220.830374] Lustre: Skipped 1 previous similar message [ 6225.067563] Lustre: server umount lustre-MDT0000 complete [ 6225.892203] LustreError: 184420:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6225.931516] LustreError: 184420:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6229.648623] LustreError: 184405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444285 with bad export cookie 8799961491443474903 [ 6229.671237] LustreError: 184405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6229.675223] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6230.504875] Lustre: server umount lustre-MDT0001 complete [ 6243.478292] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 6246.102973] Key type lgssc unregistered [ 6246.451590] LNet: 187986:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6246.470422] LNetError: 187986:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6247.532796] LNet: Removed LNI 192.168.202.136@tcp [ 6248.512566] Key type .llcrypt unregistered [ 6248.515723] Key type ._llcrypt unregistered [ 6273.820867] Key type ._llcrypt registered [ 6273.834533] Key type .llcrypt registered [ 6273.984599] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 6275.583661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6275.700028] alg: No test for adler32 (adler32-zlib) [ 6276.801048] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 6276.997481] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 6278.696950] Key type lgssc registered [ 6280.508773] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6290.254347] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6290.269314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6291.652975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6295.860625] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6301.159288] LustreError: 189400:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6301.195869] LustreError: 189400:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6305.626251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6306.250188] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6312.149780] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6322.669844] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6329.679972] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6337.440828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6337.909305] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6345.753367] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6354.016235] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6361.523188] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6370.114557] Lustre: lustre-OST0000: Not available for connect from 192.168.202.36@tcp (stopping) [ 6371.809749] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6371.816821] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6371.819909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6371.825283] Lustre: Skipped 1 previous similar message [ 6375.208941] Lustre: lustre-OST0000: Not available for connect from 192.168.202.36@tcp (stopping) [ 6375.216974] Lustre: Skipped 1 previous similar message [ 6375.380987] Lustre: server umount lustre-OST0000 complete [ 6382.492697] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9ed9105bd800.ost_server_uuid 50 [ 6384.366728] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6398.671736] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6398.684409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6403.553937] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6403.572267] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6403.587372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6405.050654] Lustre: server umount lustre-MDT0000 complete [ 6409.191533] LustreError: 190127:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6409.205725] LustreError: 191243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444465 with bad export cookie 730143658743379134 [ 6409.208741] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6409.213357] LustreError: 190127:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6409.267906] LustreError: 191243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6409.501728] Lustre: server umount lustre-MDT0001 complete [ 6421.878926] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 10:07:56 (1788444476) [ 6429.620931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6430.253526] LustreError: 193335:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6430.401489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6434.980487] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6435.815522] LustreError: 193336:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6440.933845] LustreError: 193335:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6442.340135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6442.669699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6446.656387] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6453.973502] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6459.916344] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6467.086275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6467.509360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6473.484847] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6475.699977] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6481.770537] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6488.715417] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6488.986477] Lustre: lustre-OST0001: new disk, initializing [ 6488.997472] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6489.005226] Lustre: 196176:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6489.134636] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6495.886487] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6498.391435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6498.402050] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6498.523800] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6506.063176] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6509.029781] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6509.048786] Lustre: Skipped 1 previous similar message [ 6509.050456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6509.054826] Lustre: Skipped 2 previous similar messages [ 6523.360462] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6523.493616] Lustre: server umount lustre-OST0000 complete [ 6524.407502] LustreError: 195159:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6524.431253] LustreError: 195159:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6530.962954] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6531.129976] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6535.767080] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6535.902849] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6542.637561] LustreError: 195158:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6542.653851] LustreError: 195158:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 6555.105700] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6555.106958] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6555.116889] LustreError: Skipped 1 previous similar message [ 6555.124465] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6555.133681] Lustre: Skipped 1 previous similar message [ 6555.152712] Lustre: Skipped 8 previous similar messages [ 6556.660232] Lustre: server umount lustre-OST0001 complete [ 6560.746778] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6565.355769] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6565.378438] Lustre: Skipped 1 previous similar message [ 6567.186490] Lustre: server umount lustre-MDT0000 complete [ 6570.858973] LustreError: 193360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444626 with bad export cookie 730143658743380205 [ 6570.860797] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6570.874591] LustreError: 193360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6571.137968] Lustre: server umount lustre-MDT0001 complete [ 6596.137288] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 10:10:50 (1788444650) [ 6602.517101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6602.730413] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6602.786635] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6603.002119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6607.251669] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6615.874959] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6615.995139] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6616.029296] Lustre: 199618:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6621.176414] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6629.344088] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6635.700528] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6644.350559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6644.640801] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6650.683366] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6652.322188] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6660.749303] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6667.278447] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6667.324401] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6667.532204] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6667.535809] Lustre: Skipped 2 previous similar messages [ 6673.295464] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6681.126562] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6686.689706] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6686.692818] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6686.699497] LustreError: Skipped 1 previous similar message [ 6686.702602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6686.702609] Lustre: Skipped 3 previous similar messages [ 6686.736644] Lustre: Skipped 1 previous similar message [ 6690.156714] Lustre: server umount lustre-OST0000 complete [ 6692.149561] LustreError: 201468:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6692.169756] LustreError: 201468:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 6712.866479] Lustre: server umount lustre-OST0001 complete [ 6718.947723] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6718.982914] Lustre: Skipped 1 previous similar message [ 6727.267209] Lustre: server umount lustre-MDT0000 complete [ 6729.186126] Lustre: 188708:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788444769/real 1788444769] req@000000009f6d57c0 x1875319805085568/t0(0) o13->lustre-OST0001-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788444785 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6731.655339] LustreError: 199610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444787 with bad export cookie 730143658743381612 [ 6731.656981] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6731.666229] LustreError: 199610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6731.897913] Lustre: server umount lustre-MDT0001 complete [ 6758.107916] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 10:13:32 (1788444812) [ 6777.112937] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 6790.151254] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 6831.102938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6831.338600] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6831.359313] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6831.411116] Lustre: lustre-MDT0000: new disk, initializing [ 6831.480699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6831.531816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6837.183851] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6848.405856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6848.616061] Lustre: 207718:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6848.740855] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6848.749667] Lustre: Skipped 1 previous similar message [ 6848.834556] Lustre: lustre-MDT0001: new disk, initializing [ 6849.008336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6849.024113] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6853.957794] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6865.302618] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6873.511720] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6882.280973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6883.002968] Lustre: lustre-OST0000: new disk, initializing [ 6883.010581] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6883.026738] Lustre: 209689:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6891.101366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6891.121606] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6891.327291] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6892.814390] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6906.411627] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6912.571935] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6912.996227] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6918.800294] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6919.056744] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6922.216599] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6922.227783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6922.242203] Lustre: Skipped 2 previous similar messages [ 6922.242434] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6922.271240] Lustre: Skipped 9 previous similar messages [ 6927.386717] Lustre: server umount lustre-OST0000 complete [ 6938.216892] Lustre: server umount lustre-MDT0000 complete [ 6942.693546] LustreError: 207731:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6942.731138] LustreError: 207731:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 6943.549222] LustreError: 207712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444999 with bad export cookie 730143658743383173 [ 6943.555073] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6943.560082] LustreError: 207712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6944.592765] Lustre: server umount lustre-MDT0001 complete [ 6955.893974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6962.576476] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6973.355285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6979.276695] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 6987.610776] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6994.827234] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7002.056869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7009.139905] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7018.240852] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7023.978558] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7027.518250] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 7031.992362] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7032.196485] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7034.349496] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7040.373310] Lustre: server umount lustre-OST0000 complete [ 7048.284650] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7053.290764] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7084.071291] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7087.529316] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7087.550713] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7092.560333] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7101.696412] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7101.991762] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7102.016923] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7103.790001] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7112.165209] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7112.180327] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7112.181480] LustreError: Skipped 1 previous similar message [ 7112.193417] Lustre: Skipped 4 previous similar messages [ 7112.200433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7112.210511] Lustre: Skipped 8 previous similar messages [ 7114.415804] Lustre: server umount lustre-OST0000 complete [ 7133.152147] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7133.538923] Lustre: server umount lustre-MDT0000 complete [ 7137.525779] LustreError: 211890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445193 with bad export cookie 730143658743384048 [ 7137.542126] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7138.009341] Lustre: server umount lustre-MDT0001 complete [ 7147.187456] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 10:20:01 (1788445201) [ 7154.205466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7154.684334] LustreError: 217902:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7154.704960] LustreError: 217902:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 7154.764630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7154.773908] Lustre: Skipped 6 previous similar messages [ 7159.690824] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7168.504091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7174.164713] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7182.650824] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7191.612064] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7201.340273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7210.448576] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7219.844463] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7226.571618] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7226.858606] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7232.114223] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7232.298830] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7239.916179] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7240.171289] Lustre: lustre-OST0001: new disk, initializing [ 7240.175123] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7240.182143] Lustre: 221275:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7247.626963] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7248.095689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7248.118817] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7248.317798] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7258.133242] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7263.171570] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7263.397206] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7267.876443] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7268.045455] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7273.962755] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7273.980168] LustreError: Skipped 1 previous similar message [ 7284.201808] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7284.496320] Lustre: server umount lustre-OST0001 complete [ 7291.388412] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7291.649717] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7297.366616] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7297.614294] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7327.735498] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7333.359668] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7333.382114] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7333.383990] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7335.408603] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7343.924990] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7346.305936] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7363.043296] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7363.168994] Lustre: server umount lustre-OST0001 complete [ 7383.521703] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7383.537949] Lustre: Skipped 7 previous similar messages [ 7383.547646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7383.561021] Lustre: Skipped 18 previous similar messages [ 7387.891734] Lustre: server umount lustre-OST0000 complete [ 7400.899481] Lustre: server umount lustre-MDT0000 complete [ 7406.094330] LustreError: 224551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445462 with bad export cookie 730143658743385021 [ 7406.096158] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7406.115539] LustreError: 224551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7412.541211] Lustre: server umount lustre-MDT0001 complete [ 7439.791568] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 10:24:54 (1788445494) [ 7447.699503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7448.100549] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7448.208533] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7453.677560] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7464.752198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7464.824444] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7464.849733] Lustre: 226528:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7471.173489] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7480.057889] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7486.204435] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7493.073562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7493.199699] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7499.795389] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7509.476245] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7519.899458] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7519.981878] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7526.835435] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7534.714669] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7536.664573] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7541.586861] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7541.818474] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7547.501578] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7547.757317] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7554.713284] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9ed907ce8000.ost_server_uuid 50 [ 7556.714183] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9ed907ce8000.ost_server_uuid in IDLE state after 0 sec [ 7557.838193] Lustre: Permanently deactivating lustre-OST0001 [ 7557.859301] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7562.174982] Lustre: Permanently reactivating lustre-OST0001 [ 7562.220869] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7562.226307] Lustre: Skipped 2 previous similar messages [ 7567.853523] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 7567.865906] LustreError: Skipped 5 previous similar messages [ 7582.179412] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7582.414913] Lustre: server umount lustre-OST0001 complete [ 7585.251252] LustreError: 228376:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7585.273508] LustreError: 228376:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 37 previous similar messages [ 7593.772311] Lustre: server umount lustre-OST0000 complete [ 7605.010647] Lustre: server umount lustre-MDT0000 complete [ 7609.631237] LustreError: 226520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445665 with bad export cookie 730143658743386309 [ 7609.640619] LustreError: 226520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7609.640812] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7610.133835] Lustre: server umount lustre-MDT0001 complete [ 7639.939836] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 10:28:14 (1788445694) [ 7648.367192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7648.698981] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7648.739635] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7648.753706] Lustre: Skipped 2 previous similar messages [ 7654.071426] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7663.100828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7663.269263] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7663.310174] Lustre: 233227:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7668.376479] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7676.597491] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7683.732287] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7692.041029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7692.232627] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7692.260705] Lustre: Permanently deactivating lustre-OST0000 [ 7692.287501] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7692.543689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7692.553923] Lustre: Skipped 10 previous similar messages [ 7698.745903] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7702.077824] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7706.260420] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7713.237808] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7713.305698] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7720.068714] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7720.634678] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7729.144767] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7731.778160] Lustre: Permanently reactivating lustre-OST0000 [ 7731.832317] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7731.844417] Lustre: Skipped 2 previous similar messages [ 7738.485667] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 7738.501754] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 36 seconds. I think it's dead, and I am evicting it. exp 0000000061bba6b6, cur 1788445794 deadline 1788445788 last 1788445758 [ 7738.619827] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7755.548221] Lustre: server umount lustre-OST0001 complete [ 7766.425679] Lustre: server umount lustre-OST0000 complete [ 7776.996876] Lustre: server umount lustre-MDT0000 complete [ 7780.516378] LustreError: 233222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445836 with bad export cookie 730143658743387982 [ 7780.523892] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7780.525280] LustreError: 233222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7780.766427] Lustre: server umount lustre-MDT0001 complete [ 7792.304830] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 7795.517880] Key type lgssc unregistered [ 7795.929602] LNet: 238110:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7795.955123] LNetError: 238110:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7795.976703] LNet: Removed LNI 192.168.202.136@tcp [ 7796.865713] Key type .llcrypt unregistered [ 7796.868039] Key type ._llcrypt unregistered [ 7819.436703] Key type ._llcrypt registered [ 7819.438893] Key type .llcrypt registered [ 7819.553563] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 7820.973224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7820.988233] alg: No test for adler32 (adler32-zlib) [ 7822.116776] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 7822.482869] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 7824.176268] Key type lgssc registered [ 7825.336930] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7835.924909] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7835.960716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7837.589648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7842.343606] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7847.908679] LustreError: 239570:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7847.927670] LustreError: 239570:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7850.268648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7850.433530] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7850.475845] Lustre: 239559:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7850.484633] Lustre: Permanently deactivating lustre-MDT0001 [ 7850.496328] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7850.743110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7856.161991] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7856.189532] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7865.564961] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7872.833652] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7882.104978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7882.481579] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7883.511471] LustreError: 241414:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7883.532374] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7888.880661] LustreError: 241414:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7890.319252] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7893.997810] LustreError: 241484:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7895.047204] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7897.741115] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7900.135392] LustreError: 241414:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7900.162564] LustreError: 241414:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7904.329706] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7904.491061] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7909.871397] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7909.876137] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7910.470213] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 7919.932532] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7923.525941] Lustre: Permanently reactivating lustre-MDT0001 [ 7923.584186] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7923.588108] Lustre: Skipped 2 previous similar messages [ 7926.863106] Lustre: lustre-MDT0001: Client 68c5e671-72da-4905-af20-c8861327bb98 (at 192.168.202.36@tcp) reconnecting [ 7932.579738] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7932.625924] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7932.649370] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7932.692399] LustreError: 243289:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7932.694642] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7932.723282] LustreError: 243289:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7932.723771] LustreError: 243289:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.202.136@tcp: cfg command failed: rc = -108 [ 7932.723792] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7932.723792] [ 7932.732885] LustreError: 239563:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7936.055603] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7938.002560] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7946.448449] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7946.825085] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7952.662571] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7952.957363] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7955.765187] Lustre: Permanently deactivating lustre-MDT0001 [ 7955.774856] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7955.778334] Lustre: Skipped 2 previous similar messages [ 7966.398463] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7973.866434] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7973.869443] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7973.879157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7976.424496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7976.433138] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7976.445276] Lustre: Skipped 1 previous similar message [ 7978.855129] Lustre: server umount lustre-MDT0000 complete [ 7981.543492] LustreError: 239570:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7981.577542] LustreError: 239570:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7982.494918] LustreError: 239551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446038 with bad export cookie 5437046278863310695 [ 7982.497619] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7982.506299] LustreError: 239551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7982.958077] Lustre: server umount lustre-MDT0001 complete [ 7988.822297] Lustre: server umount lustre-OST0000 complete [ 8001.425193] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 10:34:16 (1788446056) [ 8012.301274] Lustre: server umount lustre-OST0001 complete [ 8019.139927] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 8029.361696] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 8075.720165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8076.081773] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8076.089583] Lustre: Skipped 2 previous similar messages [ 8076.128822] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8076.270468] Lustre: lustre-MDT0000: new disk, initializing [ 8076.431846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8076.491993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8081.674361] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8094.082575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8094.158764] Lustre: 247857:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8094.199124] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8094.205153] Lustre: Skipped 1 previous similar message [ 8094.285976] Lustre: lustre-MDT0001: new disk, initializing [ 8094.377607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8094.425120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8094.439200] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8098.713806] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8108.499640] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8114.372602] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8120.768542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8120.997749] Lustre: lustre-OST0000: new disk, initializing [ 8121.001874] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8121.009365] Lustre: 249826:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8121.099945] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8128.106154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8128.120434] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8128.127292] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8128.290532] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8138.515718] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8140.845853] LustreError: 247866:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8150.011916] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8150.187407] Lustre: lustre-OST0001: new disk, initializing [ 8150.201596] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8150.209926] Lustre: 250939:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8150.948743] LustreError: 247866:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8150.983878] LustreError: 249834:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8156.779421] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8157.763405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8157.777725] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8157.873328] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8161.032210] LustreError: 249834:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8166.251592] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8167.939029] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8173.548844] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8173.561061] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8173.563282] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8173.563293] Lustre: Skipped 3 previous similar messages [ 8173.567299] Lustre: Skipped 2 previous similar messages [ 8173.593140] LustreError: Skipped 1 previous similar message [ 8178.677965] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8178.686535] Lustre: Skipped 2 previous similar messages [ 8183.798649] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8183.804749] Lustre: Skipped 1 previous similar message [ 8184.288174] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8184.460943] Lustre: server umount lustre-OST0001 complete [ 8188.906944] LustreError: 249810:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8188.939608] LustreError: 249810:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8204.268248] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8204.273179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8204.275572] Lustre: Skipped 1 previous similar message [ 8204.291987] Lustre: Skipped 1 previous similar message [ 8207.226290] Lustre: server umount lustre-OST0000 complete [ 8212.449385] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8212.473587] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8217.562285] Lustre: server umount lustre-MDT0000 complete [ 8221.772427] LustreError: 247850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446277 with bad export cookie 5437046278863313509 [ 8221.774491] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8221.782856] LustreError: 247850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8228.271918] Lustre: server umount lustre-MDT0001 complete [ 8241.513428] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 8244.660364] Key type lgssc unregistered [ 8245.055904] LNet: 252976:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8245.059599] LNetError: 252976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8246.120238] LNet: Removed LNI 192.168.202.136@tcp [ 8247.041534] Key type .llcrypt unregistered [ 8247.043642] Key type ._llcrypt unregistered [ 8288.507348] Key type ._llcrypt registered [ 8288.510213] Key type .llcrypt registered [ 8288.644613] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 8290.716779] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8290.779959] alg: No test for adler32 (adler32-zlib) [ 8291.930584] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8292.163868] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 8293.849505] Key type lgssc registered [ 8295.014974] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8304.663822] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8304.722700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8306.141558] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8306.203037] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8306.457172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8311.722168] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8322.062533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8322.220806] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8322.253567] Lustre: 255185:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8322.572118] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8327.959092] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8336.923784] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8343.052259] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8351.114330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8351.278625] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8351.494308] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8358.112843] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8361.074551] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8365.945058] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8422.369718] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8422.375508] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8422.403512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8422.887226] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8422.902055] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8422.927724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8427.661747] Lustre: server umount lustre-OST0000 complete [ 8432.892872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8447.179116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8447.829383] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8454.280676] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8463.015116] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8463.223345] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8463.244920] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8463.256931] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8504.297838] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8504.315938] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8504.351911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8504.356634] Lustre: Skipped 1 previous similar message [ 8508.657580] Lustre: server umount lustre-OST0000 complete [ 8514.530441] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8514.545809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8514.547640] Lustre: Skipped 2 previous similar messages [ 8514.560236] Lustre: Skipped 1 previous similar message [ 8518.588284] Lustre: server umount lustre-MDT0000 complete [ 8522.448564] LustreError: 257055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446578 with bad export cookie 10099270993169197698 [ 8522.458192] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8522.464345] LustreError: 257055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8523.098465] Lustre: server umount lustre-MDT0001 complete [ 8534.900849] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 8538.071823] Key type lgssc unregistered [ 8538.344734] LNet: 260608:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8538.349666] LNetError: 260608:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8539.370523] LNet: Removed LNI 192.168.202.136@tcp [ 8540.610380] Key type .llcrypt unregistered [ 8540.617116] Key type ._llcrypt unregistered [ 8570.254386] Key type ._llcrypt registered [ 8570.259723] Key type .llcrypt registered [ 8570.411789] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 8571.540939] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8571.617410] alg: No test for adler32 (adler32-zlib) [ 8572.680987] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8572.932917] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 8574.689073] Key type lgssc registered [ 8575.731259] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8582.991544] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8583.010477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8584.375436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8588.552877] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8594.914398] LustreError: 262030:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8594.939594] LustreError: 262030:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8597.509286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8597.767657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8602.770222] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8611.676473] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8619.098239] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8626.722570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8627.045654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8629.148970] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8633.731670] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8642.104230] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8670.177318] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8670.179969] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8670.189783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8673.251995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8673.255171] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8673.281086] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8674.618629] Lustre: server umount lustre-OST0000 complete [ 8678.369814] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8678.378981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8678.380928] Lustre: Skipped 1 previous similar message [ 8678.391137] Lustre: Skipped 2 previous similar messages [ 8683.491480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8683.502373] Lustre: Skipped 1 previous similar message [ 8684.653286] Lustre: server umount lustre-MDT0000 complete [ 8688.288526] LustreError: 262011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446744 with bad export cookie 9412894760977607096 [ 8688.292137] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8688.298236] LustreError: 262011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8688.611779] Lustre: server umount lustre-MDT0001 complete [ 8699.951862] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 8702.296195] Key type lgssc unregistered [ 8702.648700] LNet: 266334:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8702.657506] LNetError: 266334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8702.677245] LNet: Removed LNI 192.168.202.136@tcp [ 8703.435978] Key type .llcrypt unregistered [ 8703.439202] Key type ._llcrypt unregistered [ 8721.948098] Key type ._llcrypt registered [ 8721.950211] Key type .llcrypt registered [ 8722.032571] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 8723.263375] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8723.299529] alg: No test for adler32 (adler32-zlib) [ 8724.329471] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8724.676092] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 8726.372354] Key type lgssc registered [ 8727.288024] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8735.113627] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8735.139253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8736.652271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8742.637304] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8745.955718] LustreError: 267552:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8745.972231] LustreError: 267552:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8751.076565] LustreError: 267553:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8751.526202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8751.919681] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8756.339520] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8765.198091] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8772.539173] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8781.550451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8781.802264] LustreError: 269374:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8782.095477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8787.452742] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8790.582816] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8798.386309] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8811.550949] Lustre: server umount lustre-OST0000 complete [ 8817.123095] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8817.135181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8817.136933] Lustre: Skipped 1 previous similar message [ 8817.161969] Lustre: Skipped 1 previous similar message [ 8822.247040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8822.258243] Lustre: Skipped 1 previous similar message [ 8825.412159] Lustre: server umount lustre-MDT0000 complete [ 8827.364293] LustreError: 268793:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8827.385284] LustreError: 268793:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8828.384433] Lustre: 266863:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788446868/real 1788446868] req@00000000d98a3d71 x1875322370815488/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788446884 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8828.412915] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8829.503753] LustreError: 267533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446885 with bad export cookie 14324318487197828177 [ 8829.505997] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8829.515374] LustreError: 267533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8829.896177] Lustre: server umount lustre-MDT0001 complete [ 8839.796496] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 8842.685573] Key type lgssc unregistered [ 8843.011918] LNet: 271231:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8843.018572] LNetError: 271231:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8843.043583] LNet: Removed LNI 192.168.202.136@tcp [ 8843.777676] Key type .llcrypt unregistered [ 8843.781474] Key type ._llcrypt unregistered [ 8870.619866] Key type ._llcrypt registered [ 8870.624573] Key type .llcrypt registered [ 8870.748364] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 8871.900024] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8872.083591] alg: No test for adler32 (adler32-zlib) [ 8873.100039] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8873.337217] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 8875.040172] Key type lgssc registered [ 8876.260150] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8884.065407] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8884.087822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8885.589397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8890.051402] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8894.955881] LustreError: 272648:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8894.987331] LustreError: 272648:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8898.722720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8899.149400] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8903.508545] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8911.139180] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8917.682470] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8925.947282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8926.405689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8930.498993] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8932.606668] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 8939.346595] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8966.628791] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8966.644054] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8966.671073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8970.722848] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8970.729398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8970.734708] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8970.751832] Lustre: Skipped 1 previous similar message [ 8972.495681] Lustre: server umount lustre-OST0000 complete [ 8980.965262] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8980.978017] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8980.978136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8980.979744] Lustre: Skipped 1 previous similar message [ 8986.083787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8986.102209] Lustre: Skipped 2 previous similar messages [ 8990.179425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8990.185893] Lustre: Skipped 1 previous similar message [ 8992.224197] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8992.442966] Lustre: server umount lustre-MDT0000 complete [ 8995.304401] LustreError: 272649:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8995.325118] LustreError: 272649:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8996.580390] LustreError: 272629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447052 with bad export cookie 3718757786543436392 [ 8996.593487] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8996.595722] LustreError: 272629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8996.947084] Lustre: server umount lustre-MDT0001 complete [ 9009.893191] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 9012.857440] Key type lgssc unregistered [ 9013.252961] LNet: 276958:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9013.257996] LNetError: 276958:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9014.313233] LNet: Removed LNI 192.168.202.136@tcp [ 9015.230222] Key type .llcrypt unregistered [ 9015.232297] Key type ._llcrypt unregistered [ 9034.969263] Key type ._llcrypt registered [ 9034.970604] Key type .llcrypt registered [ 9035.032523] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 9035.879965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9036.109157] alg: No test for adler32 (adler32-zlib) [ 9037.256745] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 9037.761547] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 9039.520167] Key type lgssc registered [ 9040.905482] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9049.018192] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9049.037042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9050.489899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9056.202365] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9060.868364] LustreError: 278191:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9060.908583] LustreError: 278191:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9065.956862] LustreError: 278193:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9067.402119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9068.073989] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9072.792351] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9080.045787] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9085.227443] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9092.656507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9092.916419] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9095.039930] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 9099.576485] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9107.023630] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9120.741231] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9120.745713] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9120.756907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9120.766436] Lustre: Skipped 1 previous similar message [ 9120.776160] Lustre: Skipped 1 previous similar message [ 9122.857971] Lustre: server umount lustre-OST0000 complete [ 9129.445079] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9129.453254] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9129.472335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9130.978146] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9130.978907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9131.002941] Lustre: Skipped 1 previous similar message [ 9132.728478] Lustre: server umount lustre-MDT0000 complete [ 9136.103113] LustreError: 278201:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9136.116286] LustreError: 278201:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9136.654978] LustreError: 278156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447192 with bad export cookie 766295235641965085 [ 9136.666430] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9136.668737] LustreError: 278156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9136.939242] Lustre: server umount lustre-MDT0001 complete [ 9150.244629] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 9153.860655] Key type lgssc unregistered [ 9154.275529] LNet: 281891:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9154.287755] LNetError: 281891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9154.307956] LNet: Removed LNI 192.168.202.136@tcp [ 9155.542641] Key type .llcrypt unregistered [ 9155.543974] Key type ._llcrypt unregistered [ 9168.814462] Key type ._llcrypt registered [ 9168.816731] Key type .llcrypt registered [ 9168.924667] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9185.435163] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 9199.939238] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 9200.908969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9200.966289] alg: No test for adler32 (adler32-zlib) [ 9202.068754] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 9202.330952] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 9204.032184] Key type lgssc registered [ 9205.114869] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9240.382135] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9240.410460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9241.711783] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9241.779382] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9241.885383] Lustre: lustre-MDT0000: new disk, initializing [ 9241.965773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9241.982958] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9246.069768] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9255.954555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9256.102667] Lustre: 285442:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9256.157172] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9256.161730] Lustre: Skipped 1 previous similar message [ 9256.260762] Lustre: lustre-MDT0001: new disk, initializing [ 9256.387511] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9256.431533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9256.442727] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9260.790520] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9270.270518] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9275.792608] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9282.538357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9282.783515] Lustre: lustre-OST0000: new disk, initializing [ 9282.786234] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9282.791672] Lustre: 287408:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9282.867283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9289.391938] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9290.864158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9290.880152] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9291.090616] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9299.874422] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9304.982466] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9305.137236] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9311.162966] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9311.389599] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9316.835720] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9316.848757] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9316.855979] LustreError: Skipped 1 previous similar message [ 9316.856983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9316.911969] Lustre: Skipped 1 previous similar message [ 9319.344728] Lustre: server umount lustre-OST0000 complete [ 9322.977093] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9322.987380] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9323.005575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9323.007937] Lustre: Skipped 1 previous similar message [ 9327.077044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9327.079266] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9332.196972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9332.202728] Lustre: Skipped 2 previous similar messages [ 9337.313061] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9337.318908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9337.351179] Lustre: Skipped 1 previous similar message [ 9337.904215] Lustre: server umount lustre-MDT0000 complete [ 9342.095217] LustreError: 285435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447398 with bad export cookie 9019702557888183412 [ 9342.110024] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9342.372334] Lustre: server umount lustre-MDT0001 complete [ 9352.123162] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 10:56:46 (1788447406) [ 9360.218743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9361.308237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9366.501237] LustreError: 289821:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9366.540406] LustreError: 289821:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9367.146051] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9371.631876] LustreError: 289820:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9376.748524] LustreError: 289821:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9376.797692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9377.164354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9381.770492] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9389.916433] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9395.822948] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9403.162476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9403.490766] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9409.345429] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9417.836547] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9432.549507] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9432.554925] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9432.568282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9437.599783] Lustre: server umount lustre-OST0000 complete [ 9443.809525] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9443.818404] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9443.827918] LustreError: Skipped 1 previous similar message [ 9443.850715] Lustre: Skipped 2 previous similar messages [ 9447.521756] Lustre: server umount lustre-MDT0000 complete [ 9448.930583] LustreError: 289821:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9448.954600] LustreError: 289821:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9450.720580] LustreError: 289800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447506 with bad export cookie 9019702557888184294 [ 9450.728232] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9450.733095] LustreError: 289800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9451.011993] Lustre: server umount lustre-MDT0001 complete [ 9462.378177] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [ 9465.446749] Key type lgssc unregistered [ 9465.734689] LNet: 293374:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9465.742116] LNetError: 293374:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9465.760328] LNet: Removed LNI 192.168.202.136@tcp [ 9466.561151] Key type .llcrypt unregistered [ 9466.563980] Key type ._llcrypt unregistered [ 9497.008120] Key type ._llcrypt registered [ 9497.009949] Key type .llcrypt registered [ 9497.281334] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 9516.809730] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 9518.371863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9518.410377] alg: No test for adler32 (adler32-zlib) [ 9519.854829] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 9520.214526] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [ 9521.888328] Key type lgssc registered [ 9523.046082] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9581.758104] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9581.793545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9583.173838] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9583.216366] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9583.317672] Lustre: lustre-MDT0000: new disk, initializing [ 9583.403303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9583.428696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9588.513815] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9600.156479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9600.354455] Lustre: 297201:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9600.417373] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9600.420183] Lustre: Skipped 1 previous similar message [ 9600.503734] Lustre: lustre-MDT0001: new disk, initializing [ 9600.591977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9600.670898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9600.703769] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9607.644900] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9620.811718] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9628.441199] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9637.200186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9637.392984] Lustre: lustre-OST2710: new disk, initializing [ 9637.395123] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9637.398215] Lustre: 299182:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9637.456496] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9642.588410] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9647.195847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9647.207813] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9647.328723] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9652.946950] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9659.794147] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9659.979512] Lustre: lustre-OST03e8: new disk, initializing [ 9659.986485] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9659.994671] Lustre: 300245:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9660.123239] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9666.666287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9666.679878] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9666.772910] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9667.459924] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9675.877843] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9684.068962] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9684.272302] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9689.239367] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9689.434637] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9694.313883] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9694.493606] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9699.531085] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9699.802382] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9705.752426] Lustre: server umount lustre-MDT0000 complete [ 9708.002825] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9708.016849] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9708.047569] Lustre: Skipped 1 previous similar message [ 9709.711459] LustreError: 297194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447765 with bad export cookie 9407466416073950895 [ 9709.722145] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9709.722730] LustreError: 297194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9710.029719] Lustre: server umount lustre-MDT0001 complete [ 9724.843489] Lustre: server umount lustre-OST2710 complete [ 9739.165758] Lustre: server umount lustre-OST03e8 complete [ 9758.707878] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [ 9770.652629] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 9821.109260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9821.461904] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9821.531972] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9821.662872] Lustre: lustre-MDT0000: new disk, initializing [ 9821.787842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9821.820200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9826.120416] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9838.378922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9838.523584] Lustre: 305495:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9838.568833] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9838.572465] Lustre: Skipped 1 previous similar message [ 9838.656209] Lustre: lustre-MDT0001: new disk, initializing [ 9838.750038] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9838.778909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9838.789456] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9844.347932] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9855.155687] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9862.854500] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9871.278917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9871.596245] Lustre: lustre-OST0000: new disk, initializing [ 9871.607800] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9871.623318] Lustre: 307469:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9871.809285] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9878.865957] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9881.717198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9881.730115] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9881.865832] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9891.087579] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9897.552230] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9897.812405] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9902.987640] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9903.221289] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9907.681621] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9907.702996] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9907.711355] LustreError: Skipped 1 previous similar message [ 9907.712088] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9907.745636] Lustre: Skipped 2 previous similar messages [ 9912.081054] Lustre: server umount lustre-OST0000 complete [ 9917.934341] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9917.944133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9917.954098] Lustre: Skipped 1 previous similar message [ 9917.960528] Lustre: Skipped 1 previous similar message [ 9922.907221] Lustre: server umount lustre-MDT0000 complete [ 9923.050280] LustreError: 305508:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9923.085847] LustreError: 305508:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9927.148577] LustreError: 305490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447983 with bad export cookie 9407466416073952575 [ 9927.150729] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9927.168654] LustreError: 305490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9927.955734] Lustre: server umount lustre-MDT0001 complete [ 9937.064276] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 11:06:31 (1788447991) [ 9939.365602] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9941.562804] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 11:06:36 (1788447996) [ 9946.809236] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [ 9979.011291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9979.365911] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9979.390329] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9979.583346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9986.106233] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [ 9995.729428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9995.854739] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9995.898938] Lustre: 311327:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10001.114507] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10024.308620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10024.748146] Lustre: Denying initial registration attempt from nid 192.168.202.136@tcp, specified as failover [10024.762160] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [10024.775538] LustreError: 312782:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -99 [10024.790407] LustreError: 312782:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [10024.798403] LustreError: 312782:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10024.906523] Lustre: server umount lustre-OST0000 complete [10024.911354] LustreError: 312782:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [10033.633896] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10033.643806] Lustre: Skipped 1 previous similar message [10033.651466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10033.660025] Lustre: Skipped 2 previous similar messages [10038.755772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10039.256844] Lustre: server umount lustre-MDT0000 complete [10043.683346] LustreError: 311319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448099 with bad export cookie 9407466416073953457 [10043.697124] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10044.092691] Lustre: server umount lustre-MDT0001 complete [10056.977094] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [10059.754470] Key type lgssc unregistered [10060.162783] LNet: 313901:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10060.187674] LNetError: 313901:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10060.202404] LNet: Removed LNI 192.168.202.136@tcp [10061.352641] Key type .llcrypt unregistered [10061.355832] Key type ._llcrypt unregistered [10075.191441] Key type ._llcrypt registered [10075.194521] Key type .llcrypt registered [10075.324146] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [10076.417868] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10076.497705] alg: No test for adler32 (adler32-zlib) [10077.701953] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [10078.028962] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [10079.698704] Key type lgssc registered [10080.868456] Lustre: Echo OBD driver; http://www.lustre.org/ [10135.828436] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [10144.933436] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10144.952596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10146.217306] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10146.239766] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10146.363634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10151.100878] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10162.352957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10162.453338] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10162.494602] Lustre: 316664:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10162.718397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10167.775106] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10176.569742] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10182.931546] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10203.252727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10203.666187] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10203.908940] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10210.889895] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10219.247578] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10221.537690] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10221.551637] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10221.570393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10222.051277] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10222.066540] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10222.096718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10222.117747] Lustre: Skipped 1 previous similar message [10227.176024] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10232.299956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10232.311919] Lustre: Skipped 1 previous similar message [10235.360138] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10235.529287] Lustre: server umount lustre-OST0000 complete [10242.538485] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10242.559641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10242.563930] Lustre: Skipped 2 previous similar messages [10252.772845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10252.784489] Lustre: Skipped 3 previous similar messages [10254.304199] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10254.596633] Lustre: server umount lustre-MDT0000 complete [10258.783832] LustreError: 316654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448314 with bad export cookie 16936998551630315672 [10258.797376] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10259.204649] Lustre: server umount lustre-MDT0001 complete [10274.814951] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [10279.551566] Key type lgssc unregistered [10279.906971] LNet: 320242:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10279.913872] LNetError: 320242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10280.937407] LNet: Removed LNI 192.168.202.136@tcp [10282.222242] Key type .llcrypt unregistered [10282.224958] Key type ._llcrypt unregistered [10309.454733] Key type ._llcrypt registered [10309.458131] Key type .llcrypt registered [10309.526843] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [10310.392884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10310.469183] alg: No test for adler32 (adler32-zlib) [10311.827224] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [10312.060478] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [10313.712465] Key type lgssc registered [10315.325638] Lustre: Echo OBD driver; http://www.lustre.org/ [10324.393301] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10324.443057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10325.943581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10331.203513] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10335.203140] LustreError: 321656:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10335.231980] LustreError: 321656:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10340.323174] LustreError: 321655:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10340.813606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10341.110291] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10345.652520] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10353.940744] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10360.653539] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10369.642160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10370.068171] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10377.044548] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10385.969088] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10397.668096] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10397.675667] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10397.683923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10397.700562] Lustre: Skipped 2 previous similar messages [10397.707924] Lustre: Skipped 2 previous similar messages [10398.572701] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.36@tcp (stopping) [10400.941522] Lustre: server umount lustre-MDT0000 complete [10402.786498] LustreError: 321655:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10402.807075] LustreError: 321655:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10404.739530] LustreError: 321634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448460 with bad export cookie 3918908106421697923 [10404.764017] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10404.770237] LustreError: 321634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10407.905086] LustreError: 322342:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10407.921847] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10407.926075] LustreError: 322342:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10407.936808] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10409.964963] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10409.975490] Lustre: Skipped 1 previous similar message [10413.874615] LustreError: 321652:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10413.908369] LustreError: 321652:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10414.051524] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10414.070741] Lustre: Skipped 1 previous similar message [10419.168236] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10419.844468] Lustre: server umount lustre-MDT0001 complete [10429.306679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10438.408647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10450.239918] LustreError: 323487:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10450.257704] LustreError: 323487:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10450.333602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10450.411883] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10455.738965] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10465.165880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10465.529970] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10471.402260] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10473.965339] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10479.704435] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10481.674245] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10488.381749] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10489.871930] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10496.484361] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10496.487473] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10496.491884] LustreError: Skipped 1 previous similar message [10496.495986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10496.495995] Lustre: Skipped 2 previous similar messages [10496.536392] Lustre: Skipped 1 previous similar message [10502.758300] Lustre: server umount lustre-OST0000 complete [10509.801950] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10509.824613] Lustre: Skipped 1 previous similar message [10514.360850] Lustre: server umount lustre-MDT0000 complete [10514.913459] LustreError: 325252:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10514.945743] LustreError: 325252:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10518.835227] LustreError: 321635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448574 with bad export cookie 3918908106421705070 [10518.839209] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10518.847677] LustreError: 321635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10519.084718] Lustre: server umount lustre-MDT0001 complete [10532.357220] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [10535.700994] Key type lgssc unregistered [10536.135267] LNet: 327926:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10536.146877] LNetError: 327926:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10536.173530] LNet: Removed LNI 192.168.202.136@tcp [10537.048958] Key type .llcrypt unregistered [10537.051029] Key type ._llcrypt unregistered [10564.420133] Key type ._llcrypt registered [10564.422462] Key type .llcrypt registered [10564.505911] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [10565.466418] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10565.481664] alg: No test for adler32 (adler32-zlib) [10566.483133] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [10566.709284] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [10568.368431] Key type lgssc registered [10569.697102] Lustre: Echo OBD driver; http://www.lustre.org/ [10577.246858] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10577.269994] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10578.803860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10585.272374] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10588.137576] LustreError: 329340:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10588.159314] LustreError: 329340:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10593.253306] LustreError: 329339:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10597.416262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10598.208215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10604.639943] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10612.864778] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10619.624683] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10622.949395] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10622.957073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10622.963153] Lustre: Skipped 1 previous similar message [10624.036611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10624.062339] Lustre: Skipped 2 previous similar messages [10629.091630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10629.100901] Lustre: Skipped 1 previous similar message [10634.212043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10634.220251] Lustre: Skipped 1 previous similar message [10637.280181] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10637.534790] Lustre: server umount lustre-MDT0000 complete [10639.336165] LustreError: 329340:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10639.350793] LustreError: 329340:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10641.979231] LustreError: 330956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448698 with bad export cookie 18162038321076421615 [10641.997811] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10641.999702] LustreError: 330956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10642.359096] Lustre: server umount lustre-MDT0001 complete [10658.502203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10658.970460] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10659.043263] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10659.397468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10664.980966] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10675.084467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10675.216130] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10675.258443] Lustre: 331929:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10675.504830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10680.601747] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10689.362909] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10694.654917] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10704.887838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10705.159818] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10705.178106] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10705.191467] LustreError: 333813:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [10705.200215] LustreError: 333813:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [10705.208448] LustreError: 333813:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10705.317854] Lustre: server umount lustre-OST0000 complete [10705.322107] LustreError: 333813:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10716.533045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10716.713758] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10716.862085] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10723.013119] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10725.554186] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10730.036356] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10737.880164] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10738.034302] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10738.189914] Lustre: lustre-OST0001: new disk, initializing [10738.195940] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10738.212308] Lustre: 335355:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10738.337506] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10745.527062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10745.544229] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10745.635300] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10745.934462] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10754.633899] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10761.196287] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10761.215114] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10761.242124] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10764.257867] Lustre: server umount lustre-OST0001 complete [10766.308904] LustreError: 334770:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10766.324674] LustreError: 334770:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10771.428678] LustreError: 334282:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10771.444313] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10771.447739] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10771.447753] Lustre: Skipped 1 previous similar message [10771.449639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10771.449647] Lustre: Skipped 1 previous similar message [10771.454441] LustreError: 334282:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10771.531534] LustreError: Skipped 1 previous similar message [10781.680117] LustreError: 334282:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10781.699898] LustreError: 334282:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10782.179594] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10782.477824] Lustre: server umount lustre-OST0000 complete [10786.786826] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10786.818701] Lustre: Skipped 1 previous similar message [10791.907881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10791.914594] Lustre: Skipped 8 previous similar messages [10792.561746] Lustre: server umount lustre-MDT0000 complete [10796.589541] LustreError: 331920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448852 with bad export cookie 18162038321076422168 [10796.602421] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10796.793083] Lustre: server umount lustre-MDT0001 complete [10809.011228] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [10812.748813] Key type lgssc unregistered [10812.969438] LNet: 337314:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10812.973170] LNetError: 337314:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10812.993239] LNet: Removed LNI 192.168.202.136@tcp [10813.939141] Key type .llcrypt unregistered [10813.941043] Key type ._llcrypt unregistered [10888.549818] Key type ._llcrypt registered [10888.558360] Key type .llcrypt registered [10888.714251] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [10904.981286] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [10906.240134] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10906.263124] alg: No test for adler32 (adler32-zlib) [10907.409614] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [10907.598369] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [10909.250834] Key type lgssc registered [10910.242855] Lustre: Echo OBD driver; http://www.lustre.org/ [10941.340196] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10941.365474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10942.636055] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10942.685691] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10942.802183] Lustre: lustre-MDT0000: new disk, initializing [10942.864692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10942.889376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10946.898826] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10956.980421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10957.115314] Lustre: 342248:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10957.173222] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10957.180041] Lustre: Skipped 1 previous similar message [10957.248368] Lustre: lustre-MDT0001: new disk, initializing [10957.340675] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10957.375281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10957.388259] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10961.179170] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10969.510121] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10976.134481] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10983.572501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10983.786478] Lustre: lustre-OST0000: new disk, initializing [10983.790986] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10983.799585] Lustre: 344212:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10983.880947] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10990.029821] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [10992.208845] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10992.253559] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10992.382748] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11000.243842] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11005.656894] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11005.834651] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11010.149315] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11010.449414] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11013.090407] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11013.098646] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11013.102389] LustreError: Skipped 1 previous similar message [11013.103621] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11013.132140] Lustre: Skipped 1 previous similar message [11018.216192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11018.222939] Lustre: Skipped 2 previous similar messages [11018.609203] Lustre: server umount lustre-OST0000 complete [11023.330957] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11023.354754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11023.362708] Lustre: Skipped 1 previous similar message [11028.532486] Lustre: server umount lustre-MDT0000 complete [11032.627729] LustreError: 342239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449088 with bad export cookie 658167750720038974 [11032.634438] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11032.639900] LustreError: 342239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11033.026937] Lustre: server umount lustre-MDT0001 complete [11041.427719] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 11:24:56 (1788449096) [11050.120974] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 11:25:04 (1788449104) [11059.108422] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11059.927552] LustreError: 346970:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11059.952779] LustreError: 346970:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11060.069469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11065.319816] LustreError: 346971:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11065.367645] LustreError: 346971:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11065.785630] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11070.440561] LustreError: 346970:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11075.559499] LustreError: 346971:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11075.725928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11076.060561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11081.350445] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11092.525262] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11099.377933] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11107.392327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11107.814911] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11115.096440] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11123.577117] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11141.626314] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [11143.730106] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [11145.224476] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [11146.890310] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [11149.124817] Lustre: Failing over lustre-MDT0000 [11149.381576] Lustre: server umount lustre-MDT0000 complete [11150.160471] LustreError: 348513:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11150.824761] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11150.838994] Lustre: Skipped 3 previous similar messages [11160.365515] LustreError: 348513:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11160.376172] LustreError: 348513:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [11165.995735] Lustre: 340471:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788449206/real 1788449206] req@000000009c98b34b x1875324659900800/t0(0) o400->MGC192.168.202.136@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788449222 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11166.042300] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11168.971377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11176.419895] LustreError: 346970:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11176.448413] LustreError: 340470:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a23b08b8 x1875324659906176/t0(0) o250->MGC192.168.202.136@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11176.450891] LustreError: 346970:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [11176.806531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11176.849419] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11181.871636] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11182.065453] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11182.107574] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [11182.174653] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11182.502867] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11195.193165] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11197.244246] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11201.177154] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [11204.372122] Lustre: Failing over lustre-MDT0000 [11204.793243] Lustre: server umount lustre-MDT0000 complete [11207.651255] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11207.660551] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11207.696744] Lustre: Skipped 1 previous similar message [11212.586357] LustreError: 348513:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11212.602241] LustreError: 348513:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11223.008306] Lustre: 340473:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788449263/real 1788449263] req@000000009d24335e x1875324659923456/t0(0) o400->MGC192.168.202.136@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788449279 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11223.038184] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11225.627404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11233.592506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11233.662956] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11238.184843] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11238.928559] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11238.940036] Lustre: Skipped 2 previous similar messages [11238.944899] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [11239.089843] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11239.314725] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11252.010175] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11253.960676] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11264.493604] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11264.498818] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11264.518513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11264.549753] Lustre: Skipped 2 previous similar messages [11264.560873] Lustre: Skipped 2 previous similar messages [11266.493853] Lustre: server umount lustre-MDT0000 complete [11270.172283] LustreError: 346950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449326 with bad export cookie 658167750720041690 [11270.192648] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11270.195368] LustreError: 346950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11270.568944] Lustre: server umount lustre-MDT0001 complete [11285.906885] Lustre: server umount lustre-OST0000 complete [11303.984492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11304.435405] LustreError: 353337:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11304.458573] LustreError: 353337:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 22 previous similar messages [11304.564992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11309.604344] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11319.151935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11324.174027] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11333.716855] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11340.515311] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11348.602789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11350.164493] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11357.424844] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11367.525578] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11375.077686] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11375.081593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11375.097769] Lustre: Skipped 2 previous similar messages [11375.126115] Lustre: Skipped 1 previous similar message [11380.439750] Lustre: server umount lustre-OST0000 complete [11385.320381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11385.342358] Lustre: Skipped 3 previous similar messages [11390.920965] Lustre: server umount lustre-MDT0000 complete [11394.547795] LustreError: 354071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449450 with bad export cookie 658167750720042425 [11394.553942] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11394.556633] LustreError: 354071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11394.870661] Lustre: server umount lustre-MDT0001 complete [11407.380198] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [11411.039944] Key type lgssc unregistered [11411.397269] LNet: 356870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11411.403231] LNetError: 356870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11411.430871] LNet: Removed LNI 192.168.202.136@tcp [11412.378688] Key type .llcrypt unregistered [11412.380791] Key type ._llcrypt unregistered [11444.664530] Key type ._llcrypt registered [11444.666776] Key type .llcrypt registered [11444.801462] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [11446.350740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11446.504176] alg: No test for adler32 (adler32-zlib) [11447.893538] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11448.197173] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [11449.984212] Key type lgssc registered [11451.601943] Lustre: Echo OBD driver; http://www.lustre.org/ [11461.234258] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11461.247519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11462.793322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11467.745985] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11472.369870] LustreError: 358387:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11472.392274] LustreError: 358387:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11477.423512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11477.479167] LustreError: 358388:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11477.662559] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11482.075788] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11491.361627] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11498.167285] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11507.234928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11507.707509] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11512.839947] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11516.694220] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11526.169766] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11538.415611] Lustre: server umount lustre-MDT0000 complete [11539.426257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11539.453772] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11539.503340] LustreError: 358388:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11542.499400] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11542.512689] LustreError: 359424:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11542.522312] Lustre: Skipped 1 previous similar message [11542.576994] LustreError: 359424:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11543.287309] LustreError: 358368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449599 with bad export cookie 8273210763568833454 [11543.289559] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11543.301151] LustreError: 358368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11543.778550] Lustre: server umount lustre-MDT0001 complete [11558.943401] Lustre: server umount lustre-OST0000 complete [11574.467090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11574.996351] LustreError: 362102:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11575.078337] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11580.297270] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11585.524774] LustreError: 362103:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11585.546555] LustreError: 362103:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11590.559911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11591.034859] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11595.798841] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11603.797852] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11610.075091] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11617.646206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11618.010988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11620.102380] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11625.180820] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11633.774229] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11647.461237] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11647.468219] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11647.481169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11651.042977] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11651.046422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11651.077282] Lustre: Skipped 2 previous similar messages [11652.756543] Lustre: server umount lustre-MDT0000 complete [11656.163072] LustreError: 363474:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11656.190710] LustreError: 363474:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11656.952849] LustreError: 362082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449713 with bad export cookie 8273210763568834567 [11656.961462] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11656.971637] LustreError: 362082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11657.292727] Lustre: server umount lustre-MDT0001 complete [11671.618191] Lustre: server umount lustre-OST0000 complete [11689.112391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11689.530246] LustreError: 366011:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11689.602680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11694.411725] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11703.323441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11708.588467] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11716.881377] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11723.695245] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11730.929289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11731.207116] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11731.211193] Lustre: Skipped 1 previous similar message [11734.274783] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11738.373113] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11746.644848] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11753.899622] Lustre: server umount lustre-OST0000 complete [11759.074324] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11759.084850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11759.093933] Lustre: Skipped 2 previous similar messages [11759.119069] Lustre: Skipped 1 previous similar message [11764.195907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11764.201225] Lustre: Skipped 1 previous similar message [11769.340917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11769.343609] Lustre: Skipped 1 previous similar message [11770.144164] Lustre: 357697:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788449810/real 1788449810] req@00000000ef01a21e x1875325227196928/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788449826 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11770.174329] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11771.360267] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11771.620102] Lustre: server umount lustre-MDT0000 complete [11775.456710] LustreError: 365993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449831 with bad export cookie 8273210763568835589 [11775.463521] Lustre: 357696:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788449815/real 1788449815] req@0000000079f59e2a x1875325227199616/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788449831 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11775.469439] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11775.476527] LustreError: 365993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11775.487483] LustreError: 366741:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [11775.518407] Lustre: 357696:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [11775.520420] LustreError: 366741:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [11775.832349] Lustre: server umount lustre-MDT0001 complete [11786.784149] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [11789.910700] Key type lgssc unregistered [11790.169825] LNet: 369543:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11790.185170] LNetError: 369543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11791.216730] LNet: Removed LNI 192.168.202.136@tcp [11792.001088] Key type .llcrypt unregistered [11792.003324] Key type ._llcrypt unregistered [11822.581760] Key type ._llcrypt registered [11822.584742] Key type .llcrypt registered [11822.712723] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [11824.475722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11824.543327] alg: No test for adler32 (adler32-zlib) [11825.817989] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11826.097124] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [11827.840122] Key type lgssc registered [11829.198766] Lustre: Echo OBD driver; http://www.lustre.org/ [11840.069980] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11840.103993] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11840.116894] LustreError: 370981:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11840.158944] LustreError: 370981:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11840.172258] LustreError: 370981:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11840.180249] LustreError: 370981:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11840.192571] LustreError: 370981:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11849.122972] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11849.129275] LustreError: 371380:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11849.144955] LustreError: 371380:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11849.151688] LustreError: 371380:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11849.163601] LustreError: 371380:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11849.176223] LustreError: 371380:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11871.452926] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [11874.170147] Key type lgssc unregistered [11874.462928] LNet: 372383:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11874.492802] LNetError: 372383:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11874.510922] LNet: Removed LNI 192.168.202.136@tcp [11875.672573] Key type .llcrypt unregistered [11875.674054] Key type ._llcrypt unregistered [11897.673896] Key type ._llcrypt registered [11897.675535] Key type .llcrypt registered [11897.753967] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [11914.783260] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [11915.712174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11915.897427] alg: No test for adler32 (adler32-zlib) [11917.026778] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11917.356914] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [11919.040186] Key type lgssc registered [11920.242602] Lustre: Echo OBD driver; http://www.lustre.org/ [11957.003058] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11957.032229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11958.434155] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11958.501185] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11958.638534] Lustre: lustre-MDT0000: new disk, initializing [11958.730041] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11958.753837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11963.556555] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11974.382611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11974.554859] Lustre: 375538:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11974.650127] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11974.652581] Lustre: Skipped 1 previous similar message [11974.725751] Lustre: lustre-MDT0001: new disk, initializing [11974.827048] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11974.897299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11974.908948] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11979.708512] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [11989.223147] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11995.171546] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12001.820218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12002.029284] Lustre: lustre-OST0000: new disk, initializing [12002.032641] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12002.037917] Lustre: 377504:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12002.108093] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12007.636904] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12010.027038] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12010.040860] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12010.147165] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12017.413103] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12023.257195] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12023.409715] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12027.675846] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12027.842923] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12030.964667] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12030.969554] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12030.979060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12030.995048] LustreError: Skipped 1 previous similar message [12035.820872] Lustre: server umount lustre-OST0000 complete [12041.189232] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12041.200568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12041.207203] Lustre: Skipped 2 previous similar messages [12041.219615] Lustre: Skipped 2 previous similar messages [12045.943172] Lustre: server umount lustre-MDT0000 complete [12049.655371] LustreError: 375528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450105 with bad export cookie 14297806529881066171 [12049.673446] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12049.675325] LustreError: 375528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12050.060733] Lustre: server umount lustre-MDT0001 complete [12058.684446] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [12060.299842] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 11:41:55 (1788450115) [12067.316328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12067.910994] LustreError: 379964:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12067.932061] LustreError: 379964:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12068.076859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12073.448930] LustreError: 379965:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12073.473725] LustreError: 379965:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12073.944138] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12078.566111] LustreError: 379964:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12083.193240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12083.559733] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12088.505494] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12096.735725] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12102.602877] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12108.677244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12108.954714] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12115.578323] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12124.343657] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12133.084363] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12133.328809] Lustre: lustre-OST0001: new disk, initializing [12133.333346] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12133.343680] Lustre: 382804:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12133.530220] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12140.750643] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12141.675205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12141.681815] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12141.837290] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12150.928558] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12157.414041] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12157.422850] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12157.424414] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12157.446734] Lustre: Skipped 1 previous similar message [12157.458241] Lustre: Skipped 1 previous similar message [12160.886685] Lustre: server umount lustre-OST0001 complete [12162.544061] LustreError: 381788:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12162.587713] LustreError: 381788:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12167.661826] LustreError: 381787:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12167.684720] LustreError: 381787:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12177.896483] LustreError: 381786:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12177.911888] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12177.916800] LustreError: 381786:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12177.941734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12177.945783] Lustre: Skipped 2 previous similar messages [12183.013399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12183.018217] Lustre: Skipped 1 previous similar message [12183.619209] Lustre: server umount lustre-OST0000 complete [12188.133354] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12188.152512] Lustre: Skipped 2 previous similar messages [12193.250291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12193.253070] Lustre: Skipped 3 previous similar messages [12193.389599] Lustre: server umount lustre-MDT0000 complete [12196.067130] LustreError: 379944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450252 with bad export cookie 14297806529881067060 [12196.072737] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12196.076329] LustreError: 379944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12202.439142] Lustre: server umount lustre-MDT0001 complete [12211.747408] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [12214.367446] Key type lgssc unregistered [12214.712588] LNet: 384793:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12214.718378] LNetError: 384793:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12214.728458] LNet: Removed LNI 192.168.202.136@tcp [12215.476442] Key type .llcrypt unregistered [12215.477884] Key type ._llcrypt unregistered [12261.726652] Key type ._llcrypt registered [12261.728491] Key type .llcrypt registered [12261.815762] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [12262.588622] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12262.699082] alg: No test for adler32 (adler32-zlib) [12263.754819] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12263.955733] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [12265.600135] Key type lgssc registered [12266.351093] Lustre: Echo OBD driver; http://www.lustre.org/ [12273.605981] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12273.617966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12274.908769] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12274.929626] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12275.171021] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12279.580893] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12289.324735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12289.464366] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12289.521298] Lustre: 387588:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12289.797301] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12295.085706] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12301.682196] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12306.958996] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12309.985960] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12309.990455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12309.998738] Lustre: Skipped 1 previous similar message [12310.008081] Lustre: Skipped 1 previous similar message [12315.107558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12315.118151] Lustre: Skipped 1 previous similar message [12320.227250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12320.238895] Lustre: Skipped 1 previous similar message [12323.296157] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12323.610983] Lustre: server umount lustre-MDT0000 complete [12327.158754] LustreError: 387578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450383 with bad export cookie 2622426640546502323 [12327.167336] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12327.170034] LustreError: 387578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12327.542797] Lustre: server umount lustre-MDT0001 complete [12336.956845] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 11:46:32 (1788450392) [12342.515427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12342.866547] LustreError: 390099:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12342.876647] LustreError: 390099:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12342.931244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12346.223613] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12348.390928] LustreError: 390100:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12353.512088] LustreError: 390099:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12353.927539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12354.364740] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12358.428388] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12365.866219] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12371.545806] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12378.539534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12378.630106] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12378.828407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12385.337774] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12394.199383] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12403.604594] Lustre: Permanently deactivating lustre-OST0000 [12403.618725] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12414.049986] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12415.958966] LustreError: 393089:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12415.987956] LustreError: 393089:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12416.012794] LustreError: 393089:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12419.852408] LustreError: 393138:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12419.857525] LustreError: 393138:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12425.659635] Lustre: server umount lustre-OST0000 complete [12428.515824] LustreError: 393387:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12428.522454] LustreError: 393387:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12430.819286] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12430.822601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12430.840267] Lustre: Skipped 1 previous similar message [12435.938899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12435.952646] Lustre: Skipped 2 previous similar messages [12440.141935] Lustre: server umount lustre-MDT0000 complete [12441.061279] LustreError: 392948:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12441.102771] LustreError: 392948:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12445.278733] LustreError: 390079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450501 with bad export cookie 2622426640546502904 [12445.280783] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12445.296012] LustreError: 390079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12456.076186] Lustre: server umount lustre-MDT0001 complete [12465.502915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12470.241443] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12472.784296] LustreError: 394545:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12478.930234] Lustre: 394884:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.136@tcp [12480.901176] Lustre: server umount MGS complete [12492.152867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12492.637824] LustreError: 395610:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12492.662318] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12492.669238] Lustre: Skipped 1 previous similar message [12492.727930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12497.704522] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12503.010973] LustreError: 395611:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12503.033724] LustreError: 395611:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12507.910246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12508.130344] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [12508.283677] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12513.798204] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12522.892422] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12529.291376] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12531.327599] Lustre: Permanently reactivating lustre-OST0000 [12531.347257] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12531.352829] Lustre: Skipped 2 previous similar messages [12558.304875] LustreError: 397354:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12567.838528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12568.497417] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12568.504469] Lustre: Skipped 1 previous similar message [12576.880809] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12587.132814] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12593.125285] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12593.135563] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12593.168299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12593.822404] Lustre: server umount lustre-OST0000 complete [12600.812458] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12600.819648] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12609.708693] Lustre: server umount lustre-MDT0000 complete [12613.829688] LustreError: 395592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450669 with bad export cookie 2622426640546503954 [12613.841163] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12614.487384] Lustre: server umount lustre-MDT0001 complete [12627.247121] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [12630.388219] Key type lgssc unregistered [12630.687991] LNet: 399976:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12630.696511] LNetError: 399976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12630.722961] LNet: Removed LNI 192.168.202.136@tcp [12631.752133] Key type .llcrypt unregistered [12631.754703] Key type ._llcrypt unregistered [12652.774303] Key type ._llcrypt registered [12652.776830] Key type .llcrypt registered [12652.878764] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [12670.391620] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [12671.439828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12671.490096] alg: No test for adler32 (adler32-zlib) [12672.718270] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12672.981810] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [12674.656809] Key type lgssc registered [12675.880670] Lustre: Echo OBD driver; http://www.lustre.org/ [12716.854621] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12716.923554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12718.245206] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12718.309222] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12718.460987] Lustre: lustre-MDT0000: new disk, initializing [12718.642792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12718.676536] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12723.160740] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12734.037041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12734.287932] Lustre: 403134:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12734.406873] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12734.410650] Lustre: Skipped 1 previous similar message [12734.505331] Lustre: lustre-MDT0001: new disk, initializing [12734.666893] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12734.721638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12734.732703] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12739.935284] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12750.863888] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12758.025271] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12766.587688] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12766.923949] Lustre: lustre-OST0000: new disk, initializing [12766.932842] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12766.948858] Lustre: 405102:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12767.134937] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12774.316524] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12776.519708] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12776.528476] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12776.624384] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12784.745444] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12789.293647] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12789.554311] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12793.659603] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12793.825723] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12797.415840] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12797.419613] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12797.458784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12797.470089] Lustre: Skipped 1 previous similar message [12802.533913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12802.541925] Lustre: Skipped 1 previous similar message [12807.656804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12807.669657] Lustre: Skipped 1 previous similar message [12810.209147] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12810.599231] Lustre: server umount lustre-OST0000 complete [12816.866559] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12816.872698] LustreError: Skipped 1 previous similar message [12816.875877] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12816.891180] Lustre: Skipped 1 previous similar message [12816.905408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12817.891464] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12820.781860] Lustre: server umount lustre-MDT0000 complete [12824.550468] LustreError: 403126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450880 with bad export cookie 5108014193824702915 [12824.575136] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12824.589125] LustreError: 403126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12825.063780] Lustre: server umount lustre-MDT0001 complete [12833.797738] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12836.049560] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 11:54:50 (1788450890) [12843.778453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12844.408860] LustreError: 407569:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12844.438624] LustreError: 407569:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12844.488849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12849.348822] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12849.633748] LustreError: 407570:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12854.767048] LustreError: 407569:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12857.936479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12858.312393] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12862.723597] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12870.327418] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12876.267443] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12883.050256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12883.391334] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12889.473620] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [12897.204836] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12899.058890] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12901.648557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12910.051023] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12910.052832] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12910.076560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12910.093554] Lustre: Skipped 3 previous similar messages [12913.375638] Lustre: server umount lustre-OST0000 complete [12919.778175] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12919.789049] LustreError: Skipped 1 previous similar message [12919.796316] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12919.811753] Lustre: Skipped 1 previous similar message [12919.822818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12925.409944] LustreError: 407570:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12925.424456] LustreError: 407570:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12926.605798] Lustre: server umount lustre-MDT0000 complete [12930.546494] LustreError: 407569:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12930.600442] LustreError: 407569:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12931.417107] LustreError: 407550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450987 with bad export cookie 5108014193824703797 [12931.425603] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12931.435022] LustreError: 407550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12931.886871] Lustre: server umount lustre-MDT0001 complete [12944.699055] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [12948.914702] Key type lgssc unregistered [12949.462330] LNet: 411296:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12949.480547] LNetError: 411296:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12950.506127] LNet: Removed LNI 192.168.202.136@tcp [12951.853762] Key type .llcrypt unregistered [12951.856213] Key type ._llcrypt unregistered [12983.038128] Key type ._llcrypt registered [12983.040907] Key type .llcrypt registered [12983.187231] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [12983.503486] Key type .llcrypt unregistered [12983.505698] Key type ._llcrypt unregistered [13006.210177] Key type ._llcrypt registered [13006.216278] Key type .llcrypt registered [13006.487959] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [13009.112027] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13009.142761] alg: No test for adler32 (adler32-zlib) [13010.496223] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [13010.929142] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [13012.696223] Key type lgssc registered [13013.884920] Lustre: Echo OBD driver; http://www.lustre.org/ [13022.132690] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13022.150835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13023.553944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13029.076709] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13033.954376] LustreError: 413505:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13033.982318] LustreError: 413505:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13039.083305] LustreError: 413506:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13039.197096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13039.563816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13044.708118] LustreError: 413505:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13044.727690] LustreError: 413505:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13045.518413] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13049.825635] LustreError: 413506:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13049.847361] LustreError: 413506:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13049.850867] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [13053.096800] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13054.952758] LustreError: 413500:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13054.977781] LustreError: 413500:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13060.172789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13060.677358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13064.972991] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13075.426073] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13075.443214] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13075.469222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13075.937185] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13075.955546] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13075.988571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13079.689715] Lustre: server umount lustre-OST0000 complete [13086.182027] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13086.186612] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13086.199196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13086.218804] Lustre: Skipped 1 previous similar message [13086.790899] Lustre: server umount lustre-MDT0000 complete [13091.034396] LustreError: 413487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451147 with bad export cookie 6873034099662119611 [13091.041124] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13091.044223] LustreError: 413487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13091.675845] Lustre: server umount lustre-MDT0001 complete [13104.391611] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [13107.318683] Key type lgssc unregistered [13107.658157] LNet: 416653:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13107.668204] LNetError: 416653:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13107.686092] LNet: Removed LNI 192.168.202.136@tcp [13108.535423] Key type .llcrypt unregistered [13108.543635] Key type ._llcrypt unregistered [13136.403320] Key type ._llcrypt registered [13136.406334] Key type .llcrypt registered [13136.578771] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [13138.107101] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13138.229737] alg: No test for adler32 (adler32-zlib) [13139.197300] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [13139.429741] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [13141.160198] Key type lgssc registered [13143.214490] Lustre: Echo OBD driver; http://www.lustre.org/ [13152.911814] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13152.930579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13168.480689] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13173.601210] LustreError: 418042:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.136@tcp: failed processing log, type 4: rc = -110 [13200.355993] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13207.080614] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13216.451487] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13224.601815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13225.217154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13225.296093] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13230.168972] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13230.563842] LustreError: 418067:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13230.586600] LustreError: 418067:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13235.702202] LustreError: 418068:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13235.730502] LustreError: 418068:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13240.437804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13240.906519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13245.647290] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13254.430438] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13262.187260] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13271.527658] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13271.531133] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13271.535082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13271.567954] Lustre: Skipped 1 previous similar message [13275.953978] Lustre: server umount lustre-OST0000 complete [13281.763819] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13281.765953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13281.766292] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13281.766297] LustreError: Skipped 1 previous similar message [13281.780892] Lustre: Skipped 1 previous similar message [13281.814486] Lustre: Skipped 2 previous similar messages [13282.581053] Lustre: server umount lustre-MDT0000 complete [13286.447906] LustreError: 418050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451342 with bad export cookie 691789908649350960 [13286.449244] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13286.459717] LustreError: 418050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13286.740426] Lustre: server umount lustre-MDT0001 complete [13299.091188] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [13301.554153] Key type lgssc unregistered [13301.963422] LNet: 421621:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13301.976980] LNetError: 421621:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13302.002188] LNet: Removed LNI 192.168.202.136@tcp [13302.944226] Key type .llcrypt unregistered [13302.946512] Key type ._llcrypt unregistered [13332.167494] Key type ._llcrypt registered [13332.169489] Key type .llcrypt registered [13332.311761] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [13333.647233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13333.753259] alg: No test for adler32 (adler32-zlib) [13334.958498] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [13335.303512] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [13337.000655] Key type lgssc registered [13338.283987] Lustre: Echo OBD driver; http://www.lustre.org/ [13346.886715] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13346.908710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13348.457772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13353.155499] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13358.570346] LustreError: 423036:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13358.591569] LustreError: 423036:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13363.691947] LustreError: 423037:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13364.109793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13364.559289] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13369.821623] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13378.262107] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13385.714579] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13393.602388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13394.070981] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13402.171843] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13412.163766] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13418.684620] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13424.099661] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13424.103708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13424.136708] Lustre: Skipped 1 previous similar message [13424.152783] Lustre: Skipped 2 previous similar messages [13428.751482] Lustre: server umount lustre-MDT0000 complete [13429.218115] LustreError: 425155:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13429.244581] LustreError: 425155:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13432.701690] LustreError: 423018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.36@tcp arrived at 1788451488 with bad export cookie 6822298442115760474 [13432.718894] LustreError: 423018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13434.341893] LustreError: 423037:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13434.364552] LustreError: 423037:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13435.562034] LustreError: 423017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451491 with bad export cookie 6822298442115759809 [13435.578029] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13435.583613] LustreError: 423017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13436.396512] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13436.430941] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13436.445889] Lustre: Skipped 1 previous similar message [13436.453302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13436.465409] Lustre: Skipped 1 previous similar message [13439.457814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13439.459734] LustreError: 423036:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13439.499884] LustreError: 423036:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13441.854187] Lustre: server umount lustre-OST0000 complete [13449.616974] Lustre: server umount lustre-MDT0001 complete [13461.473288] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [13465.060280] Key type lgssc unregistered [13465.428949] LNet: 426720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13465.446503] LNetError: 426720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13465.466767] LNet: Removed LNI 192.168.202.136@tcp [13466.613451] Key type .llcrypt unregistered [13466.617317] Key type ._llcrypt unregistered [13498.060950] Key type ._llcrypt registered [13498.066531] Key type .llcrypt registered [13498.211043] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [13499.210069] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13499.273105] alg: No test for adler32 (adler32-zlib) [13500.812414] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [13501.061882] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [13502.768215] Key type lgssc registered [13504.514395] Lustre: Echo OBD driver; http://www.lustre.org/ [13514.236700] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13514.257777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13515.582364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13520.655164] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13524.964141] LustreError: 428135:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13524.987562] LustreError: 428135:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13529.774733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13530.076103] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13534.456614] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13542.263990] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13548.663538] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13557.438089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13557.946189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13560.017636] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13564.569630] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13573.402354] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13580.076597] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.36@tcp (stopping) [13580.773118] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13580.776712] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13580.823184] Lustre: Skipped 1 previous similar message [13583.891631] Lustre: server umount lustre-MDT0001 complete [13585.210106] LustreError: 428130:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13591.012741] LustreError: 428131:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13591.044797] LustreError: 428131:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13596.140333] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13596.164086] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13596.169183] LustreError: 428134:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13596.197983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13596.197994] Lustre: Skipped 1 previous similar message [13597.488530] Lustre: server umount lustre-OST0000 complete [13601.258809] LustreError: 429159:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13602.411833] Lustre: server umount lustre-MDT0000 complete [13621.424674] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [13624.122942] Key type lgssc unregistered [13624.368507] LNet: 431818:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13624.372325] LNetError: 431818:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13624.386936] LNet: Removed LNI 192.168.202.136@tcp [13625.245654] Key type .llcrypt unregistered [13625.247939] Key type ._llcrypt unregistered [13654.080986] Key type ._llcrypt registered [13654.083292] Key type .llcrypt registered [13654.260623] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [13671.126965] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [13672.107283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13672.296240] alg: No test for adler32 (adler32-zlib) [13673.507217] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [13673.718125] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [13675.376313] Key type lgssc registered [13676.508945] Lustre: Echo OBD driver; http://www.lustre.org/ [13714.267879] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [13732.241331] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13732.291396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13733.616660] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13733.664363] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13733.766381] Lustre: lustre-MDT0000: new disk, initializing [13733.831253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13733.843849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13738.541949] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13755.951615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13756.392964] Lustre: lustre-OST0000: new disk, initializing [13756.399392] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13756.410708] Lustre: Skipped 1 previous similar message [13756.419540] Lustre: 436852:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13756.513281] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13762.417404] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13763.176824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13763.200420] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13763.245831] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13772.729634] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13792.031027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13792.218679] Lustre: 435831:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13792.342323] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13792.491207] Lustre: lustre-MDT0001: new disk, initializing [13792.688622] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13792.760089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13792.779568] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13797.962340] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [13809.331437] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13809.641250] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13815.714367] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13816.159778] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13826.017923] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13826.028318] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13826.034957] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13826.048962] Lustre: Skipped 1 previous similar message [13826.078772] Lustre: Skipped 1 previous similar message [13826.896507] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.36@tcp (stopping) [13831.146690] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13831.152752] Lustre: Skipped 1 previous similar message [13836.256191] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13836.266534] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13836.275482] Lustre: Skipped 2 previous similar messages [13836.539707] Lustre: server umount lustre-MDT0001 complete [13842.222398] LustreError: 436496:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.36@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13842.236797] LustreError: 436496:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13846.507799] LustreError: 436496:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13846.533127] LustreError: 436496:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13851.622156] LustreError: 435843:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13851.645625] LustreError: 435843:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13856.741372] LustreError: 435842:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13860.321830] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13860.335163] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13860.364106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13861.862242] LustreError: 435843:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13871.072130] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13871.303439] Lustre: server umount lustre-OST0000 complete [13872.104993] LustreError: 435842:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13872.132234] LustreError: 435842:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13881.711382] Lustre: server umount lustre-MDT0000 complete [13899.316430] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [13903.299351] Key type lgssc unregistered [13903.784916] LNet: 440606:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13903.789148] LNetError: 440606:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13903.819530] LNet: Removed LNI 192.168.202.136@tcp [13905.199584] Key type .llcrypt unregistered [13905.204371] Key type ._llcrypt unregistered [13949.860493] Key type ._llcrypt registered [13949.868340] Key type .llcrypt registered [13950.017863] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [13950.406412] Key type .llcrypt unregistered [13950.409453] Key type ._llcrypt unregistered [13967.201800] Key type ._llcrypt registered [13967.206615] Key type .llcrypt registered [13967.378588] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [13968.796713] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13968.828252] alg: No test for adler32 (adler32-zlib) [13969.893939] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [13970.133374] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [13971.856482] Key type lgssc registered [13973.021489] Lustre: Echo OBD driver; http://www.lustre.org/ [14002.758627] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14002.785080] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14004.152864] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14004.205591] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14004.297277] Lustre: lustre-MDT0000: new disk, initializing [14004.396401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14004.416853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14009.029541] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [14020.287186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14020.565394] Lustre: 443980:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14020.667326] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14020.675381] Lustre: Skipped 1 previous similar message [14020.807386] Lustre: lustre-MDT0001: new disk, initializing [14020.935913] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14020.983178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14021.001492] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14025.185818] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [14035.279995] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14041.059894] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14048.329576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14048.562239] Lustre: lustre-OST0000: new disk, initializing [14048.565306] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14048.570359] Lustre: 445946:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14048.643624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14055.168909] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [14058.615724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14058.633326] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14058.758403] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14065.908891] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14074.348607] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14074.360302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14074.367892] Lustre: Skipped 1 previous similar message [14079.464296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14079.470814] Lustre: Skipped 4 previous similar messages [14080.277609] Lustre: server umount lustre-MDT0000 complete [14083.687895] LustreError: 444774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452139 with bad export cookie 1629014992923199346 [14083.692917] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14083.696259] LustreError: 444774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14084.163090] Lustre: server umount lustre-MDT0001 complete [14098.269889] Lustre: server umount lustre-OST0000 complete [14111.344772] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 12:16:05 (1788452165) [14121.500301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14122.206259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14127.052704] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [14127.599129] LustreError: 447977:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14127.637451] LustreError: 447977:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14132.707278] LustreError: 447976:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14135.096843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14135.372500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14140.870383] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [14148.545726] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14154.042370] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14175.032495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14175.277246] Lustre: Found index 0 for lustre-OST0000, updating log [14175.428235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14181.168220] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [14190.813171] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14193.848341] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14204.394385] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14204.400838] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14204.407610] Lustre: Skipped 1 previous similar message [14204.426140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14204.438683] Lustre: Skipped 1 previous similar message [14208.634376] Lustre: server umount lustre-OST0000 complete [14226.922547] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14226.932457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14226.934699] Lustre: Skipped 2 previous similar messages [14230.686837] Lustre: server umount lustre-MDT0000 complete [14231.073530] LustreError: 448704:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14231.106746] LustreError: 448704:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14233.315069] LNet: 1 local NIs in recovery (showing 1): 192.168.202.136@tcp [14235.028284] LustreError: 447958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452291 with bad export cookie 1629014992923200739 [14235.037518] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14235.061762] LustreError: 447958:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14235.669962] Lustre: server umount lustre-MDT0001 complete [14245.580659] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 12:18:20 (1788452300) [14870.084316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14870.537080] LustreError: 454391:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14870.567891] LustreError: 454391:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14870.603522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14875.620265] LustreError: 454392:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14876.262254] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [14880.738319] LustreError: 454391:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14885.411292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14886.059612] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14890.999458] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [14900.061753] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14906.929695] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14925.603262] LNet: 1 local NIs in recovery (showing 1): 192.168.202.136@tcp [14928.682965] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14928.940981] Lustre: Found index 0 for lustre-OST0000, updating log [14929.232792] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14936.208154] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [14944.110645] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14947.499635] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14947.512461] Lustre: Skipped 1 previous similar message [14947.519627] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14952.735689] Lustre: server umount lustre-OST0000 complete [14956.965832] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14956.981846] Lustre: Skipped 1 previous similar message [14956.981926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14956.991314] Lustre: Skipped 1 previous similar message [14958.067155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14958.083807] Lustre: Skipped 1 previous similar message [14960.164521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14960.171072] Lustre: Skipped 3 previous similar messages [14964.258074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14964.264698] Lustre: Skipped 7 previous similar messages [14970.336580] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14970.544659] Lustre: server umount lustre-MDT0000 complete [14971.426780] LustreError: 454392:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14971.437592] LustreError: 454392:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14975.327650] LustreError: 454371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453031 with bad export cookie 1629014992923201824 [14975.330689] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14975.357675] LustreError: 454371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14975.891683] Lustre: server umount lustre-MDT0001 complete [14984.997833] LNet: 1 local NIs in recovery (showing 1): 192.168.202.136@tcp [15017.411460] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [15028.876641] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [15066.433374] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 12:32:01 (1788453121) [15084.330793] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [15087.085580] Key type lgssc unregistered [15087.349273] LNet: 461006:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15087.361320] LNetError: 461006:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15189.800732] LNet: Removed LNI 192.168.202.136@tcp [15190.520852] Key type .llcrypt unregistered [15190.524767] Key type ._llcrypt unregistered [15207.360738] Key type ._llcrypt registered [15207.367799] Key type .llcrypt registered [15207.500649] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [15209.127440] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15209.166287] alg: No test for adler32 (adler32-zlib) [15210.481189] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [15210.831834] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [15212.576285] Key type lgssc registered [15214.738171] Lustre: Echo OBD driver; http://www.lustre.org/ [15220.434725] LNet: Added LNI 192.168.202.136@tcp100 [8/256/0/180] [15221.372651] LNet: Added LNI 192.168.202.136@tcp101 [8/256/0/180] [15222.594446] LNet: Added LNI 192.168.202.136@tcp102 [8/256/0/180] [15223.750489] LNet: Added LNI 192.168.202.136@tcp103 [8/256/0/180] [15225.064582] LNet: Added LNI 192.168.202.136@tcp104 [8/256/0/180] [15226.367284] LNet: Added LNI 192.168.202.136@tcp105 [8/256/0/180] [15227.788421] LNet: Added LNI 192.168.202.136@tcp106 [8/256/0/180] [15229.239043] LNet: Added LNI 192.168.202.136@tcp107 [8/256/0/180] [15230.390667] LNet: Added LNI 192.168.202.136@tcp108 [8/256/0/180] [15231.478949] LNet: Added LNI 192.168.202.136@tcp109 [8/256/0/180] [15233.712446] LNet: Added LNI 192.168.202.136@tcp110 [8/256/0/180] [15235.006771] LNet: Added LNI 192.168.202.136@tcp111 [8/256/0/180] [15236.124795] LNet: Added LNI 192.168.202.136@tcp112 [8/256/0/180] [15237.527177] LNet: Added LNI 192.168.202.136@tcp113 [8/256/0/180] [15238.820870] LNet: Added LNI 192.168.202.136@tcp114 [8/256/0/180] [15240.212930] LNet: Added LNI 192.168.202.136@tcp115 [8/256/0/180] [15241.684169] LNet: Added LNI 192.168.202.136@tcp116 [8/256/0/180] [15243.019362] LNet: Added LNI 192.168.202.136@tcp117 [8/256/0/180] [15243.948132] LNet: Added LNI 192.168.202.136@tcp118 [8/256/0/180] [15244.718551] LNet: Added LNI 192.168.202.136@tcp119 [8/256/0/180] [15245.488738] LNet: Added LNI 192.168.202.136@tcp120 [8/256/0/180] [15246.408796] LNet: Added LNI 192.168.202.136@tcp121 [8/256/0/180] [15247.586223] LNet: Added LNI 192.168.202.136@tcp122 [8/256/0/180] [15248.583673] LNet: Added LNI 192.168.202.136@tcp123 [8/256/0/180] [15249.502057] LNet: Added LNI 192.168.202.136@tcp124 [8/256/0/180] [15250.322756] LNet: Added LNI 192.168.202.136@tcp125 [8/256/0/180] [15251.534563] LNet: Added LNI 192.168.202.136@tcp126 [8/256/0/180] [15252.651672] LNet: Added LNI 192.168.202.136@tcp127 [8/256/0/180] [15253.724176] LNet: Added LNI 192.168.202.136@tcp128 [8/256/0/180] [15254.691922] LNet: Added LNI 192.168.202.136@tcp129 [8/256/0/180] [15255.933577] LNet: Added LNI 192.168.202.136@tcp130 [8/256/0/180] [15256.982802] LNet: Added LNI 192.168.202.136@tcp131 [8/256/0/180] [15257.768592] LNet: Added LNI 192.168.202.136@tcp132 [8/256/0/180] [15258.866697] LNet: Added LNI 192.168.202.136@tcp133 [8/256/0/180] [15259.596698] LNet: Added LNI 192.168.202.136@tcp134 [8/256/0/180] [15260.507447] LNet: Added LNI 192.168.202.136@tcp135 [8/256/0/180] [15267.706869] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15267.738809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15268.901850] Lustre: 464034:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.136@tcp: can't add NID 192.168.202.136@tcp130: rc = -75 [15268.986790] LNetError: 461493:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.136 [15269.006046] LNetError: Protocol error connecting to 192.168.202.136@tcp116 at host 192.168.202.136:988: is it running a compatible version of Lustre? [15269.017653] LNetError: Skipped 6 previous similar messages [15269.161438] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15269.183282] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15269.216732] LNet: 7 local NIs in recovery (showing 7): 192.168.202.136@tcp123, 192.168.202.136@tcp127, 192.168.202.136@tcp126, 192.168.202.136@tcp130, 192.168.202.136@tcp124, 192.168.202.136@tcp116, 192.168.202.136@tcp129 [15269.246353] Lustre: lustre-MDT0000: new disk, initializing [15269.306476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15269.317270] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15270.259556] LNetError: 461491:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.136 [15270.276022] LNetError: 461491:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 16 previous similar messages [15270.290709] LNetError: Protocol error connecting to 192.168.202.136@tcp126 at host 192.168.202.136:988: is it running a compatible version of Lustre? [15270.309814] LNetError: Skipped 8 previous similar messages [15271.330931] LNetError: 464054:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.136 [15271.336327] LNetError: 464054:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 4 previous similar messages [15271.340810] LNetError: Protocol error connecting to 192.168.202.136@tcp131 at host 192.168.202.136:988: is it running a compatible version of Lustre? [15273.844887] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [15276.386645] LNetError: 464056:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.136 [15276.400800] LNetError: 464056:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 2 previous similar messages [15276.414381] LNetError: Protocol error connecting to 192.168.202.136@tcp123 at host 192.168.202.136:988: is it running a compatible version of Lustre? [15276.426296] LNetError: Skipped 2 previous similar messages [15284.313437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15284.496212] Lustre: 464051:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15284.587337] LNetError: 464061:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.136 [15284.590143] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15284.596747] LNetError: Protocol error connecting to 192.168.202.136@tcp123 at host 192.168.202.136:988: is it running a compatible version of Lustre? [15284.611226] Lustre: Skipped 1 previous similar message [15284.728226] Lustre: lustre-MDT0001: new disk, initializing [15284.781334] Lustre: 464852:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.136@tcp: can't add NID 192.168.202.136@tcp130: rc = -75 [15284.798077] Lustre: 464852:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15284.953830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15285.007197] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15285.027991] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15288.448875] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [15297.874458] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15299.941382] LNetError: 464054:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.136 [15299.965831] LNetError: 464054:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 12 previous similar messages [15299.978381] LNetError: Protocol error connecting to 192.168.202.136@tcp134 at host 192.168.202.136:988: is it running a compatible version of Lustre? [15300.002923] LNetError: Skipped 14 previous similar messages [15305.081928] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15312.264440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15312.588370] Lustre: lustre-OST0000: new disk, initializing [15312.594052] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15312.609531] Lustre: 466025:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15312.733912] Lustre: 466030:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.136@tcp: can't add NID 192.168.202.136@tcp130: rc = -75 [15312.747685] Lustre: 466030:0:(lustre_peer.c:124:class_add_uuid()) Skipped 17 previous similar messages [15312.804769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15316.389677] LNetError: 461490:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.136 [15316.401450] LNetError: 461490:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 17 previous similar messages [15316.419138] LNetError: Protocol error connecting to 192.168.202.136@tcp115 at host 192.168.202.136:988: is it running a compatible version of Lustre? [15316.438947] LNetError: Skipped 15 previous similar messages [15320.133033] Lustre: 466364:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.136@tcp: can't add NID 192.168.202.136@tcp130: rc = -75 [15320.164974] Lustre: 466364:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15320.232285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15320.242422] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [15320.251534] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15320.396491] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15329.644018] LNet: 2 local NIs in recovery (showing 2): 192.168.202.136@tcp134, 192.168.202.136@tcp123 [15330.054223] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15341.031576] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15341.032796] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15341.043958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15341.070288] Lustre: Skipped 1 previous similar message [15346.152092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15346.160499] Lustre: Skipped 2 previous similar messages [15351.268042] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15351.270236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15351.289379] Lustre: Skipped 1 previous similar message [15351.596643] Lustre: server umount lustre-OST0000 complete [15354.285307] LNetError: 464054:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.136 [15354.296496] LNetError: 464054:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 32 previous similar messages [15354.302752] LNetError: Protocol error connecting to 192.168.202.136@tcp133 at host 192.168.202.136:988: is it running a compatible version of Lustre? [15354.315596] LNetError: Skipped 32 previous similar messages [15356.389115] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15356.397806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15356.405668] Lustre: Skipped 1 previous similar message [15356.422277] Lustre: Skipped 1 previous similar message [15361.536815] Lustre: server umount lustre-MDT0000 complete [15365.500735] LustreError: 464043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453421 with bad export cookie 12194775147852471375 [15365.510951] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15365.513775] LustreError: 464043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15366.013571] Lustre: server umount lustre-MDT0001 complete [15378.235859] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [15383.200777] Key type lgssc unregistered [15383.914334] LNet: 467940:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15383.928031] LNetError: 467940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15383.945289] LNet: Removed LNI 192.168.202.136@tcp [15384.992595] LNet: Removed LNI 192.168.202.136@tcp100 [15384.994955] LNetError: 467940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15384.999244] LNetError: 467940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15385.003897] LNet: Removed LNI 192.168.202.136@tcp101 [15386.016855] LNet: Removed LNI 192.168.202.136@tcp102 [15386.019409] LNetError: 467940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15386.030996] LNetError: 467940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15386.044982] LNet: Removed LNI 192.168.202.136@tcp103 [15387.104236] LNet: Removed LNI 192.168.202.136@tcp104 [15388.128166] LNet: Removed LNI 192.168.202.136@tcp105 [15388.134472] LNetError: 467940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15388.145974] LNetError: 467940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15389.217729] LNet: Removed LNI 192.168.202.136@tcp106 [15390.241262] LNet: Removed LNI 192.168.202.136@tcp107 [15391.264176] LNet: Removed LNI 192.168.202.136@tcp108 [15392.288388] LNet: Removed LNI 192.168.202.136@tcp109 [15392.290636] LNetError: 467940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15392.308382] LNetError: 467940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [15392.331157] LNet: Removed LNI 192.168.202.136@tcp110 [15393.376285] LNet: Removed LNI 192.168.202.136@tcp111 [15394.400319] LNet: Removed LNI 192.168.202.136@tcp112 [15395.424392] LNet: Removed LNI 192.168.202.136@tcp113 [15396.448163] LNet: Removed LNI 192.168.202.136@tcp114 [15397.472572] LNet: Removed LNI 192.168.202.136@tcp115 [15398.496196] LNet: Removed LNI 192.168.202.136@tcp116 [15399.520449] LNet: Removed LNI 192.168.202.136@tcp117 [15400.544805] LNet: Removed LNI 192.168.202.136@tcp118 [15400.552239] LNetError: 467940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15400.558529] LNetError: 467940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15400.576934] LNet: Removed LNI 192.168.202.136@tcp119 [15401.632916] LNet: Removed LNI 192.168.202.136@tcp120 [15402.656328] LNet: Removed LNI 192.168.202.136@tcp121 [15403.681841] LNet: Removed LNI 192.168.202.136@tcp122 [15403.684804] LNet: Removed LNI 192.168.202.136@tcp123 [15404.704273] LNet: Removed LNI 192.168.202.136@tcp124 [15405.728158] LNet: Removed LNI 192.168.202.136@tcp125 [15406.755200] LNet: Removed LNI 192.168.202.136@tcp126 [15407.776217] LNet: Removed LNI 192.168.202.136@tcp127 [15408.800257] LNet: Removed LNI 192.168.202.136@tcp128 [15409.824324] LNet: Removed LNI 192.168.202.136@tcp129 [15410.848272] LNet: Removed LNI 192.168.202.136@tcp130 [15411.873490] LNet: Removed LNI 192.168.202.136@tcp131 [15412.896255] LNet: Removed LNI 192.168.202.136@tcp132 [15413.920122] LNet: Removed LNI 192.168.202.136@tcp133 [15413.922527] LNet: Removed LNI 192.168.202.136@tcp134 [15414.955526] LNet: Removed LNI 192.168.202.136@tcp135 [15415.840718] Key type .llcrypt unregistered [15415.846833] Key type ._llcrypt unregistered [15440.279719] Key type ._llcrypt registered [15440.282753] Key type .llcrypt registered [15440.375607] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [15454.673717] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [15455.781639] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15455.871702] alg: No test for adler32 (adler32-zlib) [15457.040781] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [15457.266395] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [15458.930804] Key type lgssc registered [15459.951054] Lustre: Echo OBD driver; http://www.lustre.org/ [15492.119362] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 12:39:07 (1788453547) [15512.982971] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [15516.370663] Key type lgssc unregistered [15516.773865] LNet: 472026:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15516.778882] LNetError: 472026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15516.790836] LNet: Removed LNI 192.168.202.136@tcp [15517.570930] Key type .llcrypt unregistered [15517.580722] Key type ._llcrypt unregistered [15537.658674] Key type ._llcrypt registered [15537.661820] Key type .llcrypt registered [15537.882978] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [15540.119586] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15540.387230] alg: No test for adler32 (adler32-zlib) [15541.755487] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [15542.086762] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [15543.920668] Key type lgssc registered [15545.319673] Lustre: Echo OBD driver; http://www.lustre.org/ [15558.361536] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15558.392681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15559.973813] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15560.000164] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15560.171764] Lustre: lustre-MDT0000: new disk, initializing [15560.462417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15560.764353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15566.558805] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [15577.869290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15578.000435] Lustre: 473238:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15578.066493] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15578.070108] Lustre: Skipped 1 previous similar message [15578.146924] Lustre: lustre-MDT0001: new disk, initializing [15578.328337] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15578.368391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15578.380176] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15583.797601] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [15595.181959] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15602.889368] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15612.309576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15612.612198] Lustre: lustre-OST0000: new disk, initializing [15612.616692] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15612.631231] Lustre: 475206:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15612.775138] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15620.899071] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [15622.727195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15622.736170] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15622.866222] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15630.905737] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15635.572518] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing lnet_if_list [15640.723585] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15640.755582] LNet: Added LNI 192.168.202.136@tcp10 [8/256/0/180] [15645.923193] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15645.950704] LNet: Added LNI 192.168.202.136@tcp11 [8/256/0/180] [15650.553423] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15650.571834] LNet: Added LNI 192.168.202.136@tcp12 [8/256/0/180] [15655.343609] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15655.375524] LNet: Added LNI 192.168.202.136@tcp13 [8/256/0/180] [15659.633406] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15659.656916] LNet: Added LNI 192.168.202.136@tcp14 [8/256/0/180] [15665.081185] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15665.121293] LNet: Added LNI 192.168.202.136@tcp15 [8/256/0/180] [15670.429821] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15670.477220] LNet: Added LNI 192.168.202.136@tcp16 [8/256/0/180] [15676.370224] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15676.420752] LNet: Added LNI 192.168.202.136@tcp17 [8/256/0/180] [15682.164631] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15682.198128] LNet: Added LNI 192.168.202.136@tcp18 [8/256/0/180] [15687.582942] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15687.633256] LNet: Added LNI 192.168.202.136@tcp19 [8/256/0/180] [15693.199959] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15693.247645] LNet: Added LNI 192.168.202.136@tcp20 [8/256/0/180] [15698.910548] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15698.933746] LNet: Added LNI 192.168.202.136@tcp21 [8/256/0/180] [15705.471916] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15705.508524] LNet: Added LNI 192.168.202.136@tcp22 [8/256/0/180] [15712.335254] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15712.360474] LNet: Added LNI 192.168.202.136@tcp23 [8/256/0/180] [15720.680134] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15720.712233] LNet: Added LNI 192.168.202.136@tcp24 [8/256/0/180] [15728.079344] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15728.117883] LNet: Added LNI 192.168.202.136@tcp25 [8/256/0/180] [15735.559582] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15735.617184] LNet: Added LNI 192.168.202.136@tcp26 [8/256/0/180] [15744.319512] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15744.414970] LNet: Added LNI 192.168.202.136@tcp27 [8/256/0/180] [15751.816618] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15751.851319] LNet: Added LNI 192.168.202.136@tcp28 [8/256/0/180] [15757.983849] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15758.005850] LNet: Added LNI 192.168.202.136@tcp29 [8/256/0/180] [15764.261558] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15764.319524] LNet: Added LNI 192.168.202.136@tcp30 [8/256/0/180] [15770.906599] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15770.935844] LNet: Added LNI 192.168.202.136@tcp31 [8/256/0/180] [15777.226757] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15777.257658] LNet: Added LNI 192.168.202.136@tcp32 [8/256/0/180] [15783.833769] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15783.864657] LNet: Added LNI 192.168.202.136@tcp33 [8/256/0/180] [15791.060395] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15791.082433] LNet: Added LNI 192.168.202.136@tcp34 [8/256/0/180] [15797.726791] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15797.754439] LNet: Added LNI 192.168.202.136@tcp35 [8/256/0/180] [15804.798301] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15804.821846] LNet: Added LNI 192.168.202.136@tcp36 [8/256/0/180] [15811.726694] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15811.753722] LNet: Added LNI 192.168.202.136@tcp37 [8/256/0/180] [15818.709655] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15818.743342] LNet: Added LNI 192.168.202.136@tcp38 [8/256/0/180] [15825.167379] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15825.213449] LNet: Added LNI 192.168.202.136@tcp39 [8/256/0/180] [15833.056771] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15833.076032] LNet: Added LNI 192.168.202.136@tcp40 [8/256/0/180] [15840.859406] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15840.893141] LNet: Added LNI 192.168.202.136@tcp41 [8/256/0/180] [15849.145419] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15849.179159] LNet: Added LNI 192.168.202.136@tcp42 [8/256/0/180] [15857.249763] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15857.279311] LNet: Added LNI 192.168.202.136@tcp43 [8/256/0/180] [15865.668469] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15865.715659] LNet: Added LNI 192.168.202.136@tcp44 [8/256/0/180] [15875.573640] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15875.612581] LNet: Added LNI 192.168.202.136@tcp45 [8/256/0/180] [15886.452377] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15886.482900] LNet: Added LNI 192.168.202.136@tcp46 [8/256/0/180] [15897.981741] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15898.005492] LNet: Added LNI 192.168.202.136@tcp47 [8/256/0/180] [15909.097549] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15909.120812] LNet: Added LNI 192.168.202.136@tcp48 [8/256/0/180] [15918.841662] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15918.856772] LNet: Added LNI 192.168.202.136@tcp49 [8/256/0/180] [15930.718394] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15930.748376] LNet: Added LNI 192.168.202.136@tcp50 [8/256/0/180] [15952.358648] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15952.365974] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15952.380585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15955.938354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15955.941819] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15955.945084] Lustre: Skipped 1 previous similar message [15955.963890] Lustre: Skipped 1 previous similar message [15958.251611] Lustre: server umount lustre-MDT0000 complete [15963.714876] LustreError: 475205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454019 with bad export cookie 14493278569537314485 [15963.729608] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15963.743733] LustreError: 475205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15964.169508] Lustre: server umount lustre-MDT0001 complete [15982.560325] Lustre: 472509:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788454022/real 1788454022] req@000000003110aae5 x1875329520158464/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788454038 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15982.619226] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15984.608902] Lustre: 472508:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788454024/real 1788454024] req@0000000072119333 x1875329520158592/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788454040 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15987.680289] Lustre: 472507:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788454027/real 1788454027] req@00000000c4a77714 x1875329520158720/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788454043 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15989.728268] Lustre: 472510:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788454029/real 1788454029] req@00000000cfe7eadd x1875329520158976/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788454045 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15990.666676] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [15994.849476] Lustre: 472509:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788454034/real 1788454034] req@0000000018fa7e87 x1875329520159104/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788454050 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16001.273383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16008.303134] LustreError: 472506:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000002c7df5dc x1875329520159872/t0(0) o250->MGC192.168.202.136@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16008.745126] LustreError: 475199:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16008.774461] LustreError: 475199:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [16008.863040] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16011.941043] LustreError: 475200:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16011.945332] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16012.906557] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [16012.963272] LustreError: 475199:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16018.414291] LustreError: 475200:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16023.525295] LustreError: 475199:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16025.270229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16025.576359] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16027.645630] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16031.395238] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [16043.074592] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16052.462625] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16073.705088] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16073.716417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16073.722821] Lustre: Skipped 1 previous similar message [16073.737975] Lustre: Skipped 2 previous similar messages [16078.841679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16078.849845] Lustre: Skipped 1 previous similar message [16079.823260] Lustre: server umount lustre-OST0000 complete [16087.010925] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16087.025099] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16087.049095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16090.808548] Lustre: server umount lustre-MDT0000 complete [16094.179833] LustreError: 499447:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16094.212097] LustreError: 499447:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [16094.634971] LustreError: 473229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454150 with bad export cookie 14493278569537317138 [16094.639410] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16094.645676] LustreError: 473229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16094.938185] Lustre: server umount lustre-MDT0001 complete [16106.626724] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [16109.348690] Key type lgssc unregistered [16109.725922] LNet: 502254:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16109.736594] LNetError: 502254:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16109.745209] LNet: Removed LNI 192.168.202.136@tcp [16109.748789] LNet: Removed LNI 192.168.202.136@tcp10 [16109.752900] LNet: Removed LNI 192.168.202.136@tcp11 [16109.755383] LNet: Removed LNI 192.168.202.136@tcp12 [16109.760333] LNet: Removed LNI 192.168.202.136@tcp13 [16109.768753] LNet: Removed LNI 192.168.202.136@tcp14 [16109.779459] LNet: Removed LNI 192.168.202.136@tcp15 [16109.788717] LNet: Removed LNI 192.168.202.136@tcp16 [16109.800686] LNet: Removed LNI 192.168.202.136@tcp17 [16109.804213] LNet: Removed LNI 192.168.202.136@tcp18 [16109.811207] LNet: Removed LNI 192.168.202.136@tcp19 [16109.821257] LNet: Removed LNI 192.168.202.136@tcp20 [16109.832489] LNet: Removed LNI 192.168.202.136@tcp21 [16109.837089] LNet: Removed LNI 192.168.202.136@tcp22 [16109.840180] LNet: Removed LNI 192.168.202.136@tcp23 [16109.843201] LNet: Removed LNI 192.168.202.136@tcp24 [16109.845611] LNet: Removed LNI 192.168.202.136@tcp25 [16109.848516] LNet: Removed LNI 192.168.202.136@tcp26 [16109.851021] LNet: Removed LNI 192.168.202.136@tcp27 [16109.854452] LNet: Removed LNI 192.168.202.136@tcp28 [16109.856787] LNet: Removed LNI 192.168.202.136@tcp29 [16109.859691] LNet: Removed LNI 192.168.202.136@tcp30 [16109.862456] LNet: Removed LNI 192.168.202.136@tcp31 [16109.865858] LNet: Removed LNI 192.168.202.136@tcp32 [16109.875317] LNet: Removed LNI 192.168.202.136@tcp33 [16109.883962] LNet: Removed LNI 192.168.202.136@tcp34 [16109.890883] LNet: Removed LNI 192.168.202.136@tcp35 [16109.896674] LNet: Removed LNI 192.168.202.136@tcp36 [16109.900621] LNet: Removed LNI 192.168.202.136@tcp37 [16109.907496] LNet: Removed LNI 192.168.202.136@tcp38 [16109.914865] LNet: Removed LNI 192.168.202.136@tcp39 [16109.935437] LNet: Removed LNI 192.168.202.136@tcp40 [16109.937693] LNet: Removed LNI 192.168.202.136@tcp41 [16109.943176] LNet: Removed LNI 192.168.202.136@tcp42 [16109.951977] LNet: Removed LNI 192.168.202.136@tcp43 [16109.961194] LNet: Removed LNI 192.168.202.136@tcp44 [16109.969292] LNet: Removed LNI 192.168.202.136@tcp45 [16109.971580] LNet: Removed LNI 192.168.202.136@tcp46 [16109.973529] LNet: Removed LNI 192.168.202.136@tcp47 [16109.990089] LNet: Removed LNI 192.168.202.136@tcp48 [16109.992219] LNet: Removed LNI 192.168.202.136@tcp49 [16110.026746] LNet: Removed LNI 192.168.202.136@tcp50 [16110.872317] Key type .llcrypt unregistered [16110.874544] Key type ._llcrypt unregistered [16132.899397] Key type ._llcrypt registered [16132.901584] Key type .llcrypt registered [16132.973253] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [16154.380835] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [16155.501785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16155.562216] alg: No test for adler32 (adler32-zlib) [16156.698346] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [16157.034205] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [16158.777343] Key type lgssc registered [16160.418729] Lustre: Echo OBD driver; http://www.lustre.org/ [16201.501384] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16201.571258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16202.879990] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16202.941542] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16203.104372] Lustre: lustre-MDT0000: new disk, initializing [16203.303671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16203.331053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16208.256877] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [16219.974902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16220.180574] Lustre: 505448:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16220.269527] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16220.276393] Lustre: Skipped 1 previous similar message [16220.479578] Lustre: lustre-MDT0001: new disk, initializing [16220.622886] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16220.718153] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16220.732085] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16226.109485] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [16237.108491] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16243.885852] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16251.420932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16251.627469] Lustre: *** cfs_fail_loc=90f, val=0*** [16251.749819] Lustre: lustre-OST0000: new disk, initializing [16251.754235] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16251.760089] Lustre: 507464:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16251.886595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16257.942511] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [16261.207719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16261.219682] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16261.380301] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16268.613096] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16282.084334] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16282.109411] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16282.111882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16282.125581] LustreError: Skipped 1 previous similar message [16282.156605] Lustre: Skipped 1 previous similar message [16283.946260] Lustre: server umount lustre-OST0000 complete [16292.328438] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16292.329300] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16292.335309] Lustre: Skipped 1 previous similar message [16292.340090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16292.348127] Lustre: Skipped 1 previous similar message [16294.235751] Lustre: server umount lustre-MDT0000 complete [16298.484878] LustreError: 505441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454354 with bad export cookie 12838852444974527646 [16298.501103] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16299.181276] Lustre: server umount lustre-MDT0001 complete [16311.014675] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [16314.982936] Key type lgssc unregistered [16315.349049] LNet: 509368:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16315.365567] LNetError: 509368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16315.379229] LNet: Removed LNI 192.168.202.136@tcp [16316.756453] Key type .llcrypt unregistered [16316.758689] Key type ._llcrypt unregistered [16343.979848] Key type ._llcrypt registered [16343.983256] Key type .llcrypt registered [16344.215471] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [16346.273653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16346.312105] alg: No test for adler32 (adler32-zlib) [16347.860379] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [16348.607911] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [16350.544781] Key type lgssc registered [16352.760425] Lustre: Echo OBD driver; http://www.lustre.org/ [16361.939208] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16361.958940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16363.681594] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16369.766518] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [16373.223382] LustreError: 510636:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16373.242492] LustreError: 510636:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [16378.343166] LustreError: 510637:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16378.594400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16378.847130] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16383.504701] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [16392.364847] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16399.053937] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16406.328281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16406.735283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16413.749585] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [16422.483461] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16433.434600] loop0: detected capacity change from 0 to 204800000 [16433.743697] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16433.752526] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16433.777417] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16441.878663] loop0: detected capacity change from 0 to 204800000 [16442.142719] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16442.173075] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16442.190744] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16449.138041] loop0: detected capacity change from 0 to 204800000 [16457.393233] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16457.586857] Lustre: fs15246-OST0000: new disk, initializing [16457.590942] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16457.601962] Lustre: 514234:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16457.796252] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16468.832225] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [16480.191990] loop1: detected capacity change from 0 to 204800000 [16486.497387] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16486.635578] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16486.760179] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16486.887342] Lustre: fs15246-MDT0000: new disk, initializing [16487.171108] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16487.200337] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16487.332140] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16488.356853] Lustre: 515356:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000004ebf38a5 x1875330365180544/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788454560 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16491.974353] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [16497.451510] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16497.469522] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16497.495067] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16517.119151] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16517.136855] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16520.800923] Lustre: server umount fs15246-MDT0000 complete [16522.223943] LustreError: 511639:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16527.345647] LustreError: 510633:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16531.121226] Lustre: server umount fs15246-OST0000 complete [16542.178132] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16542.183120] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16542.195272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16547.811291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16547.819773] Lustre: Skipped 1 previous similar message [16548.010389] Lustre: server umount lustre-OST0000 complete [16552.931079] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16552.933230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16552.964338] Lustre: Skipped 2 previous similar messages [16558.055932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16558.076719] Lustre: Skipped 2 previous similar messages [16566.240248] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16566.592798] Lustre: server umount lustre-MDT0000 complete [16568.293315] LustreError: 510636:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16568.328591] LustreError: 510636:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [16571.425584] LustreError: 510616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454627 with bad export cookie 8340436027884528304 [16571.433377] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16571.435787] LustreError: 510616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [16572.381266] Lustre: server umount lustre-MDT0001 complete [16587.063606] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [16590.730551] Key type lgssc unregistered [16591.139689] LNet: 518455:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16591.146533] LNetError: 518455:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16591.172042] LNet: Removed LNI 192.168.202.136@tcp [16592.343432] Key type .llcrypt unregistered [16592.350911] Key type ._llcrypt unregistered [16611.828652] Key type ._llcrypt registered [16611.833458] Key type .llcrypt registered [16612.013282] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [16613.228803] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16613.330348] alg: No test for adler32 (adler32-zlib) [16614.559789] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [16614.848527] LNet: Added LNI 192.168.202.136@tcp [8/256/0/180] [16616.624431] Key type lgssc registered [16617.669336] Lustre: Echo OBD driver; http://www.lustre.org/ [16669.577785] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [16685.427578] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [16730.538737] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16730.592406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16731.939338] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16731.966565] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16732.055521] Lustre: lustre-MDT0000: new disk, initializing [16732.151581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16732.183669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16737.328575] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [16748.939146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16749.037449] Lustre: 522945:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16749.075276] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16749.078633] Lustre: Skipped 1 previous similar message [16749.187338] Lustre: lustre-MDT0001: new disk, initializing [16749.320426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16749.371118] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16749.391398] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16754.910177] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [16766.101365] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16771.907771] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16779.143415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16779.427646] Lustre: lustre-OST0000: new disk, initializing [16779.440217] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16779.445431] Lustre: 524915:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16779.576143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16785.474802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16785.489414] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16785.609070] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16786.780609] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_default_debug -1 all [16797.475982] Lustre: DEBUG MARKER: oleg236-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16803.112790] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16803.295452] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16808.885592] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16809.080748] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16811.489298] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16811.489955] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16811.494690] LustreError: Skipped 1 previous similar message [16811.497280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16811.505276] Lustre: Skipped 1 previous similar message [16811.517116] Lustre: Skipped 1 previous similar message [16816.610244] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16816.620322] Lustre: Skipped 1 previous similar message [16817.582511] Lustre: server umount lustre-OST0000 complete [16821.729519] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16821.755700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16826.338835] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16826.366889] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16826.411391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16828.420400] Lustre: server umount lustre-MDT0000 complete [16833.079374] LustreError: 522938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454889 with bad export cookie 16361021725044520016 [16833.091824] LustreError: MGC192.168.202.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16833.098461] LustreError: 522938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16833.515515] Lustre: server umount lustre-MDT0001 complete [16865.862455] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing set_hostid [16881.471434] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing load_modules_local [16917.351762] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16615 sec ========= 13:02:51 (1788454971) [16920.002261] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 13:02:54 (1788454974) === [16924.516234] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 13:02:58 (1788454978) === [16970.282910] Lustre: DEBUG MARKER: oleg236-server.virtnet: executing unload_modules_local [16973.687779] Key type lgssc unregistered [16974.258611] LNet: 531595:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16974.265000] LNetError: 531595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16974.280916] LNet: Removed LNI 192.168.202.136@tcp [16975.484131] Key type .llcrypt unregistered [16975.487911] Key type ._llcrypt unregistered