[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 453119185 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002173] x2apic enabled [ 0.002972] Switched APIC routing to physical x2apic. [ 0.003953] kvm-guest: setup PV IPIs [ 0.005842] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006017] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007009] pid_max: default: 32768 minimum: 301 [ 0.008068] LSM: Security Framework initializing [ 0.008948] Yama: becoming mindful. [ 0.009025] SELinux: Initializing. [ 0.010071] *** VALIDATE selinux *** [ 0.018735] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.022630] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.023119] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.024145] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025112] *** VALIDATE tmpfs *** [ 0.026437] *** VALIDATE proc *** [ 0.027282] *** VALIDATE cgroup *** [ 0.028009] *** VALIDATE cgroup2 *** [ 0.030173] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.031155] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.032007] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.033026] Spectre V2 : User space: Vulnerable [ 0.034008] Speculative Store Bypass: Vulnerable [ 0.037199] debug: unmapping init [mem 0xffffffffb4059000-0xffffffffb4060fff] [ 0.039179] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.040729] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.041023] ... version: 2 [ 0.042010] ... bit width: 48 [ 0.043012] ... generic registers: 4 [ 0.043945] ... value mask: 0000ffffffffffff [ 0.044013] ... max period: 00007fffffffffff [ 0.045015] ... fixed-purpose events: 3 [ 0.046014] ... event mask: 000000070000000f [ 0.047290] rcu: Hierarchical SRCU implementation. [ 0.049185] smp: Bringing up secondary CPUs ... [ 0.050392] x86: Booting SMP configuration: [ 0.051025] .... node #0, CPUs: #1 #2 #3 [ 0.054020] smp: Brought up 1 node, 4 CPUs [ 0.056013] smpboot: Max logical packages: 1 [ 0.057012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.124713] node 0 deferred pages initialised in 65ms [ 0.127008] devtmpfs: initialized [ 0.127791] x86/mm: Memory block size: 128MB [ 0.129015] gcov: version magic: 0x41383552 [ 0.131204] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.133090] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.134338] pinctrl core: initialized pinctrl subsystem [ 0.136135] [ 0.136495] ************************************************************* [ 0.138010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.139006] ** ** [ 0.141008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.142009] ** ** [ 0.144013] ** This means that this kernel is built to expose internal ** [ 0.146009] ** IOMMU data structures, which may compromise security on ** [ 0.147006] ** your system. ** [ 0.148008] ** ** [ 0.150008] ** If you see this message and you are not debugging the ** [ 0.152010] ** kernel, report this immediately to your vendor! ** [ 0.153007] ** ** [ 0.155007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.156009] ************************************************************* [ 0.158600] NET: Registered protocol family 16 [ 0.160338] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.162046] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.165064] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.169010] cpuidle: using governor menu [ 0.170740] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.173313] PCI: Using configuration type 1 for base access [ 0.176136] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.186122] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.189124] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.192154] cryptd: max_cpu_qlen set to 1000 [ 0.195252] ACPI: Added _OSI(Module Device) [ 0.197017] ACPI: Added _OSI(Processor Device) [ 0.198012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.200014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.205540] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.212745] ACPI: Interpreter enabled [ 0.215099] ACPI: PM: (supports S0 S3 S4 S5) [ 0.217017] ACPI: Using IOAPIC for interrupt routing [ 0.219111] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.223480] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.234632] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.237051] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.240020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.243082] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.247299] acpiphp: Slot [2] registered [ 0.249146] acpiphp: Slot [5] registered [ 0.250130] acpiphp: Slot [6] registered [ 0.252128] acpiphp: Slot [7] registered [ 0.253123] acpiphp: Slot [8] registered [ 0.255121] acpiphp: Slot [9] registered [ 0.256114] acpiphp: Slot [10] registered [ 0.258127] acpiphp: Slot [3] registered [ 0.259155] acpiphp: Slot [4] registered [ 0.261100] acpiphp: Slot [11] registered [ 0.262090] acpiphp: Slot [12] registered [ 0.264165] acpiphp: Slot [13] registered [ 0.265118] acpiphp: Slot [14] registered [ 0.267114] acpiphp: Slot [15] registered [ 0.269100] acpiphp: Slot [16] registered [ 0.270127] acpiphp: Slot [17] registered [ 0.271104] acpiphp: Slot [18] registered [ 0.273132] acpiphp: Slot [19] registered [ 0.275131] acpiphp: Slot [20] registered [ 0.277121] acpiphp: Slot [21] registered [ 0.278112] acpiphp: Slot [22] registered [ 0.280124] acpiphp: Slot [23] registered [ 0.282114] acpiphp: Slot [24] registered [ 0.283099] acpiphp: Slot [25] registered [ 0.285110] acpiphp: Slot [26] registered [ 0.286114] acpiphp: Slot [27] registered [ 0.288125] acpiphp: Slot [28] registered [ 0.289164] acpiphp: Slot [29] registered [ 0.291108] acpiphp: Slot [30] registered [ 0.292104] acpiphp: Slot [31] registered [ 0.294082] PCI host bridge to bus 0000:00 [ 0.295019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.297022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.300022] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.302021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.305024] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.308029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.310193] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.313071] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.316337] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.326895] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.332547] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.335018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.336014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.338030] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.341382] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.343848] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.347047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.350868] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.356017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.367962] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.374015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.379497] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.387021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.393012] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.411019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.425203] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.442018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.448017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.464018] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.472583] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.479018] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.487015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.498017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.510901] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.519016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.526014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.540020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.550969] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.557017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.562015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.576022] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.585742] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.592017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.603016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.636032] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.661254] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.663377] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.665349] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.669256] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.673303] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.678030] iommu: Default domain type: Passthrough [ 0.680440] SCSI subsystem initialized [ 0.681133] ACPI: bus type USB registered [ 0.683114] usbcore: registered new interface driver usbfs [ 0.685137] usbcore: registered new interface driver hub [ 0.687098] usbcore: registered new device driver usb [ 0.689207] pps_core: LinuxPPS API ver. 1 registered [ 0.691012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.695061] PTP clock support registered [ 0.698085] EDAC MC: Ver: 3.0.0 [ 0.700468] PCI: Using ACPI for IRQ routing [ 0.702382] NetLabel: Initializing [ 0.703000] NetLabel: domain hash size = 128 [ 0.705015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.707085] NetLabel: unlabeled traffic allowed by default [ 0.710109] vgaarb: loaded [ 0.712328] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.714012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.723208] clocksource: Switched to clocksource kvm-clock [ 0.822338] VFS: Disk quotas dquot_6.6.0 [ 0.823477] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.825857] *** VALIDATE ramfs *** [ 0.827111] *** VALIDATE hugetlbfs *** [ 0.828990] pnp: PnP ACPI init [ 0.831951] pnp: PnP ACPI: found 6 devices [ 0.847745] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.850656] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.852244] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.853766] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.855982] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.857975] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.860185] NET: Registered protocol family 2 [ 0.862896] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.868228] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.871784] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.876937] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.879778] TCP: Hash tables configured (established 65536 bind 65536) [ 0.882774] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.885192] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.887637] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.890429] NET: Registered protocol family 1 [ 0.893104] RPC: Registered named UNIX socket transport module. [ 0.895182] RPC: Registered udp transport module. [ 0.896820] RPC: Registered tcp transport module. [ 0.898444] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.900579] NET: Registered protocol family 44 [ 0.902265] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.904477] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.906387] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.908871] PCI: CLS 0 bytes, default 64 [ 0.910412] Unpacking initramfs... [ 2.384787] debug: unmapping init [mem 0xffff8eccfcc54000-0xffff8eccfffbffff] [ 2.388964] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.390683] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.393055] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.900682] Initialise system trusted keyrings [ 2.902652] Key type blacklist registered [ 2.904696] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.914286] zbud: loaded [ 2.917490] *** VALIDATE nfs *** [ 2.919048] *** VALIDATE nfs4 *** [ 2.920838] pstore: using deflate compression [ 2.923913] Platform Keyring initialized [ 3.016894] NET: Registered protocol family 38 [ 3.018542] Key type asymmetric registered [ 3.019516] Asymmetric key parser 'x509' registered [ 3.020797] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.022885] io scheduler mq-deadline registered [ 3.023939] io scheduler kyber registered [ 3.025168] io scheduler bfq registered [ 3.026414] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.028430] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.030400] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.032311] ACPI: Power Button [PWRF] [ 3.035968] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.042921] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.056127] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.062405] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.078141] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.105815] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.134781] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.140349] Non-volatile memory driver v1.3 [ 3.141656] Linux agpgart interface v0.103 [ 3.175694] virtio_blk virtio1: [vda] 149408 512-byte logical blocks (76.5 MB/73.0 MiB) [ 3.178504] vda: detected capacity change from 0 to 76496896 [ 3.193347] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.196462] vdb: detected capacity change from 0 to 1073741824 [ 3.211354] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.214812] vdc: detected capacity change from 0 to 2621440000 [ 3.230555] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.233674] vdd: detected capacity change from 0 to 2621440000 [ 3.246973] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.249275] vde: detected capacity change from 0 to 4294967296 [ 3.267767] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.269985] vdf: detected capacity change from 0 to 4294967296 [ 3.277259] libphy: Fixed MDIO Bus: probed [ 3.294457] usbcore: registered new interface driver usbserial_generic [ 3.296369] usbserial: USB Serial support registered for generic [ 3.298179] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.301454] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.302835] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.304760] mousedev: PS/2 mouse device common for all mice [ 3.307545] rtc_cmos 00:05: RTC can wake from S4 [ 3.310214] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.312884] rtc_cmos 00:05: registered as rtc0 [ 3.314776] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.316982] intel_pstate: CPU model not supported [ 3.319249] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.321767] hid: raw HID events driver (C) Jiri Kosina [ 3.324311] usbcore: registered new interface driver usbhid [ 3.326066] usbhid: USB HID core driver [ 3.326317] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.327240] drop_monitor: Initializing network drop monitor service [ 3.332368] Initializing XFRM netlink socket [ 3.333928] NET: Registered protocol family 10 [ 3.336396] Segment Routing with IPv6 [ 3.338100] NET: Registered protocol family 17 [ 3.340387] mpls_gso: MPLS GSO support [ 3.346572] RAS: Correctable Errors collector initialized. [ 3.348645] AVX version of gcm_enc/dec engaged. [ 3.350599] AES CTR mode by8 optimization enabled [ 3.422594] sched_clock: Marking stable (3422570385, 0)->(4266638479, -844068094) [ 3.426181] registered taskstats version 1 [ 3.427521] Loading compiled-in X.509 certificates [ 3.428765] zswap: loaded using pool lzo/zbud [ 3.453631] Key type big_key registered [ 3.464528] Key type encrypted registered [ 3.465815] ima: No TPM chip found, activating TPM-bypass! [ 3.467169] ima: Allocated hash algorithm: sha1 [ 3.468470] ima: No architecture policies found [ 3.469680] evm: Initialising EVM extended attributes: [ 3.471066] evm: security.selinux [ 3.471893] evm: security.ima [ 3.472583] evm: security.capability [ 3.473416] evm: HMAC attrs: 0x1 [ 3.475136] rtc_cmos 00:05: setting system clock to 2026-08-22 09:49:13 UTC (1787392153) [ 3.479460] debug: unmapping init [mem 0xffffffffb5003000-0xffffffffb51fffff] [ 3.481556] debug: unmapping init [mem 0xffffffffb3d82000-0xffffffffb4058fff] [ 3.490092] Write protecting the kernel read-only data: 28672k [ 3.492656] debug: unmapping init [mem 0xffffffffb2403000-0xffffffffb25fffff] [ 3.494907] debug: unmapping init [mem 0xffffffffb2d14000-0xffffffffb2dfffff] [ 3.528700] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.535455] systemd[1]: Detected virtualization kvm. [ 3.536869] systemd[1]: Detected architecture x86-64. [ 3.538201] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.565317] systemd[1]: No hostname configured. [ 3.567268] systemd[1]: Set hostname to . [ 3.569422] random: systemd: uninitialized urandom read (16 bytes read) [ 3.572088] systemd[1]: Initializing machine ID from random generator. [ 3.630209] random: ln: uninitialized urandom read (6 bytes read) [ 3.731489] random: systemd: uninitialized urandom read (16 bytes read) [ 3.734407] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.739654] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.748985] systemd[1]: Starting Journal Service... Starting Journal Service... [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. Starting Setup Virtual Console... [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.435092] device-mapper: uevent: version 1.0.3 [ 4.437577] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.173313] random: fast init done [ 5.183952] virtio_net virtio0 ens2: renamed from eth0 [ 5.298824] scsi host0: ata_piix [ 5.309261] scsi host1: ata_piix [ 5.310838] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.313180] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.759476] dracut-initqueue[585]: RTNETLINK answers: File exists [ 9.978297] random: crng init done [ 9.979542] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.536682] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.774730] printk: systemd: 22 output lines suppressed due to ratelimiting [ 12.058780] SELinux: Disabled at runtime. [ 12.128061] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.136578] systemd[1]: Detected virtualization kvm. [ 12.138443] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.687541] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.691166] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.697526] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.703301] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.708195] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.716977] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.727628] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Remount Root and Kernel File Systems... [ OK ] Reached target Paths. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Kernel Socket. Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... [ OK ] Stopped target Initrd File Systems. [ 12.867845] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. Starting udev Coldplug all Devices... [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.190531] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.561853] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.605217] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.668867] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.685375] EDAC sbridge: Ver: 1.1.2 [ 15.164824] Key type dns_resolver registered [ 15.503801] NFS: Registering the id_resolver key type [ 15.505788] Key type id_resolver registered [ 15.507266] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg233-server login: [ 45.027533] libcfs: loading out-of-tree module taints kernel. [ 45.074151] Key type ._llcrypt registered [ 45.075371] Key type .llcrypt registered [ 45.177367] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 66.409395] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 68.238257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 68.266958] alg: No test for adler32 (adler32-zlib) [ 69.889730] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 70.910196] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 72.705073] Key type lgssc registered [ 74.892530] Lustre: Echo OBD driver; http://www.lustre.org/ [ 95.133436] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 135.028043] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 151.639774] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 151.716039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 153.142143] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 153.180473] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 153.346278] Lustre: lustre-MDT0000: new disk, initializing [ 153.530754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 153.559989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 159.263419] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 173.539278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 173.697608] Lustre: 6507:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 173.749260] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 173.752237] Lustre: Skipped 1 previous similar message [ 173.829714] Lustre: lustre-MDT0001: new disk, initializing [ 173.920590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 173.955141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 173.966364] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 179.939815] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 186.805993] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 201.167854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 201.384389] Lustre: lustre-OST0000: new disk, initializing [ 201.387445] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 201.392157] Lustre: 8444:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 201.453886] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 207.931490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 207.950484] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 208.077138] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 209.487137] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 225.477674] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 225.689505] Lustre: lustre-OST0001: new disk, initializing [ 225.695381] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 225.706601] Lustre: 9516:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 225.792128] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 232.250320] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 233.583992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 233.605110] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 233.767602] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 244.634080] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 252.368916] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 259.936342] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing check_logdir /tmp/testlogs/ [ 263.716224] hrtimer: interrupt took 2215417 ns [ 267.444490] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing yml_node [ 274.002410] Lustre: DEBUG MARKER: Client: 2.17.54.109 [ 277.719140] Lustre: DEBUG MARKER: MDS: 2.17.54.109 [ 280.808290] Lustre: DEBUG MARKER: OSS: 2.17.54.109 [ 282.692298] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 22 05:53:51 EDT 2026 [ 300.678923] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 302.617491] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 320.993808] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 321.001314] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 321.004851] Lustre: Skipped 2 previous similar messages [ 325.417331] Lustre: server umount lustre-MDT0000 complete [ 329.812123] LustreError: 6497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392479 with bad export cookie 5448799740351825159 [ 329.825910] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 329.830035] LustreError: 6497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 331.236126] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 331.241360] LustreError: 6511:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 331.251427] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 331.251437] Lustre: Skipped 1 previous similar message [ 331.263642] Lustre: Skipped 4 previous similar messages [ 331.316418] LustreError: 6511:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 334.244197] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 334.244855] LustreError: 8656:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 334.252106] Lustre: Skipped 1 previous similar message [ 334.265944] LustreError: 8656:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 336.453926] Lustre: server umount lustre-MDT0001 complete [ 341.002468] Lustre: server umount lustre-OST0000 complete [ 344.824427] Lustre: server umount lustre-OST0001 complete [ 351.798895] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 362.815788] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 402.556940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 402.799911] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 402.830714] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 402.960668] Lustre: lustre-MDT0000: new disk, initializing [ 403.129623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 403.154740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 408.371388] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 419.205732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 419.315775] Lustre: 15070:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 419.372382] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 419.386458] Lustre: Skipped 1 previous similar message [ 419.452924] Lustre: lustre-MDT0001: new disk, initializing [ 419.541443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 419.584384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 419.600096] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 424.376226] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 433.740771] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 439.913429] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 447.707918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 448.242341] Lustre: lustre-OST0000: new disk, initializing [ 448.245794] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 448.257811] Lustre: 17037:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 448.360130] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 449.423280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 449.430769] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 449.648131] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 455.398565] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 465.012621] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 470.132983] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 470.359678] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 475.597475] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 475.815572] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 480.240504] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 480.243767] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 480.247112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 480.255404] Lustre: Skipped 1 previous similar message [ 480.268140] LustreError: Skipped 1 previous similar message [ 484.154313] Lustre: server umount lustre-OST0000 complete [ 490.476670] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 490.480891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 490.488861] Lustre: Skipped 1 previous similar message [ 490.507128] Lustre: Skipped 2 previous similar messages [ 495.031813] Lustre: server umount lustre-MDT0000 complete [ 495.586177] LustreError: 15081:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 495.618259] LustreError: 15081:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 500.463404] LustreError: 15062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392650 with bad export cookie 5448799740351826692 [ 500.487874] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 501.057517] Lustre: server umount lustre-MDT0001 complete [ 507.795749] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 05:57:35 (1787392655) [ 511.096285] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 513.011511] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 05:57:41 (1787392661) [ 522.284084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 522.958151] LustreError: 19448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 523.077210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 528.354577] LustreError: 19449:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 528.411820] LustreError: 19449:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 529.048600] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 538.596739] LustreError: 19448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 538.615525] LustreError: 19448:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 540.213057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 546.225786] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 556.167412] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 563.136330] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 572.185991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 580.076886] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 589.122261] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 603.616452] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 603.621856] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 603.624207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 603.636730] Lustre: Skipped 1 previous similar message [ 608.766086] LustreError: 21271:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 608.869243] Lustre: server umount lustre-OST0000 complete [ 613.859698] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 613.872023] Lustre: Skipped 1 previous similar message [ 624.101601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 624.111550] Lustre: Skipped 6 previous similar messages [ 628.191333] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 628.554145] Lustre: server umount lustre-MDT0000 complete [ 633.302611] LustreError: 20183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787392783 with bad export cookie 5448799740351827574 [ 633.314333] LustreError: 20183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 633.317786] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 634.134155] Lustre: server umount lustre-MDT0001 complete [ 649.181545] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 652.684349] Key type lgssc unregistered [ 653.048932] LNet: 23003:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 653.083134] LNetError: 23003:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 654.117708] LNet: Removed LNI 192.168.202.133@tcp [ 655.166168] Key type .llcrypt unregistered [ 655.168144] Key type ._llcrypt unregistered [ 685.372295] Key type ._llcrypt registered [ 685.376485] Key type .llcrypt registered [ 685.519452] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 687.253937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 687.286222] alg: No test for adler32 (adler32-zlib) [ 688.633379] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 688.937943] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 690.615376] Key type lgssc registered [ 691.921273] Lustre: Echo OBD driver; http://www.lustre.org/ [ 702.041495] loop: module loaded [ 702.236804] loop0: detected capacity change from 0 to 204800000 [ 702.758777] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 702.772669] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 703.319702] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 712.544901] loop0: detected capacity change from 0 to 204800000 [ 712.903567] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 712.917482] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 712.933531] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 719.361677] loop0: detected capacity change from 0 to 204800000 [ 726.410594] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 726.438626] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 727.856325] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 727.961186] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 728.114301] Lustre: lustre-MDT0000: new disk, initializing [ 728.302814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 728.336922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 733.636120] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 744.155828] loop1: detected capacity change from 0 to 204800000 [ 750.115908] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 750.528850] Lustre: lustre-OST07c6: new disk, initializing [ 750.538478] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 750.547090] Lustre: Skipped 1 previous similar message [ 750.565993] Lustre: 26209:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 750.680803] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 756.367690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 756.378091] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 756.453753] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 758.642413] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 763.721618] Lustre: 25196:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 763.729065] Lustre: 25196:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 763.734863] Lustre: 25196:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 763.750751] Lustre: 25196:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 763.767589] Lustre: 25196:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 763.778019] Lustre: 25196:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 785.274269] Lustre: 26217:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 785.282246] Lustre: 26217:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 785.291109] Lustre: 26217:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 785.298145] Lustre: 26217:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 785.303521] Lustre: 26217:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 785.308067] Lustre: 26217:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 790.181906] Lustre: 25195:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 790.198589] Lustre: 25195:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 790.210526] Lustre: 25195:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 790.217748] Lustre: 25195:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 790.235359] Lustre: 25195:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 790.251762] Lustre: 25195:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 790.270174] Lustre: 25195:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 790.290438] Lustre: 25195:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 790.299455] Lustre: 25195:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 790.309871] Lustre: 25195:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 790.320707] Lustre: 25195:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 790.328449] Lustre: 25195:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 793.583035] Lustre: 25196:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 793.600340] Lustre: 25196:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 793.609083] Lustre: 25196:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 793.619537] Lustre: 25196:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 793.630498] Lustre: 25196:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 793.638209] Lustre: 25196:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 793.646568] Lustre: 25196:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 793.656067] Lustre: 25196:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 793.670834] Lustre: 25196:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 793.680360] Lustre: 25196:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 793.690643] Lustre: 25196:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 793.700360] Lustre: 25196:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 799.184353] Lustre: 26217:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 799.205570] Lustre: 26217:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 799.216757] Lustre: 26217:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 799.229140] Lustre: 26217:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 799.247408] Lustre: 26217:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 799.258022] Lustre: 26217:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 799.266787] Lustre: 26217:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 799.276117] Lustre: 26217:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 799.282062] Lustre: 26217:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 799.287508] Lustre: 26217:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 799.293561] Lustre: 26217:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 799.299144] Lustre: 26217:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 807.336435] Lustre: 25197:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 807.343450] Lustre: 25197:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 807.351373] Lustre: 25197:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 807.359565] Lustre: 25197:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 807.372159] Lustre: 25197:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 807.381601] Lustre: 25197:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 807.386423] Lustre: 25197:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 807.392200] Lustre: 25197:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 807.407173] Lustre: 25197:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 807.413874] Lustre: 25197:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 807.422755] Lustre: 25197:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 807.433547] Lustre: 25197:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 863.717620] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 863.745109] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 865.484378] Lustre: server umount lustre-OST07c6 complete [ 878.645535] Lustre: server umount lustre-MDT0000 complete [ 898.583243] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 902.116918] Key type lgssc unregistered [ 902.599507] LNet: 28767:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 902.607456] LNetError: 28767:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 902.630838] LNet: Removed LNI 192.168.202.133@tcp [ 903.783648] Key type .llcrypt unregistered [ 903.786858] Key type ._llcrypt unregistered [ 927.283840] Key type ._llcrypt registered [ 927.286242] Key type .llcrypt registered [ 927.367313] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 928.629580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 928.876557] alg: No test for adler32 (adler32-zlib) [ 930.148090] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 930.583374] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 932.399294] Key type lgssc registered [ 933.806964] Lustre: Echo OBD driver; http://www.lustre.org/ [ 943.091582] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 943.114510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 944.684453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 949.748691] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 954.855224] LustreError: 30040:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 954.875250] LustreError: 30040:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 959.969952] LustreError: 30041:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 961.753835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 962.423219] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 967.644343] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 978.067407] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 987.032747] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 995.504742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 995.817540] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1003.476149] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1011.197866] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1014.262108] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1032.672967] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1032.680948] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1032.703032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1034.723147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1034.723828] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1037.723367] Lustre: server umount lustre-OST0000 complete [ 1044.448754] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1044.466834] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1044.494920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1044.502482] Lustre: Skipped 1 previous similar message [ 1049.113312] Lustre: server umount lustre-MDT0000 complete [ 1050.080688] LustreError: 30040:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1050.123365] LustreError: 30040:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1053.937439] LustreError: 30021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393203 with bad export cookie 10459355929648550536 [ 1053.938225] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1053.945570] LustreError: 30021:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1054.661741] Lustre: server umount lustre-MDT0001 complete [ 1067.972499] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 1071.495450] Key type lgssc unregistered [ 1072.081679] LNet: 33780:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1072.091067] LNetError: 33780:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1072.116808] LNet: Removed LNI 192.168.202.133@tcp [ 1073.039241] Key type .llcrypt unregistered [ 1073.041266] Key type ._llcrypt unregistered [ 1097.871753] Key type ._llcrypt registered [ 1097.884857] Key type .llcrypt registered [ 1098.162368] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1099.992741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1100.176154] alg: No test for adler32 (adler32-zlib) [ 1101.448025] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 1101.909452] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 1103.703301] Key type lgssc registered [ 1105.220189] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1114.563443] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1114.588810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1116.126429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1120.985803] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1126.371859] LustreError: 35051:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1126.402462] LustreError: 35051:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1130.929429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1131.276514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1136.736823] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1145.281336] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1152.010643] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1159.556360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1159.969725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1161.033669] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1167.492359] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1175.892651] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1191.904133] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1191.910682] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1191.912705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1191.919390] Lustre: Skipped 1 previous similar message [ 1191.942455] Lustre: Skipped 1 previous similar message [ 1194.883664] Lustre: server umount lustre-OST0000 complete [ 1202.157733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1202.174175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1202.192298] Lustre: Skipped 1 previous similar message [ 1202.217114] Lustre: Skipped 1 previous similar message [ 1205.884982] Lustre: server umount lustre-MDT0000 complete [ 1207.268202] LustreError: 35047:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1207.300024] LustreError: 35047:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1210.723730] LustreError: 35030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393360 with bad export cookie 9047407212633314934 [ 1210.725363] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1210.734587] LustreError: 35030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1211.271554] Lustre: server umount lustre-MDT0001 complete [ 1224.155408] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 1227.627503] Key type lgssc unregistered [ 1227.858116] LNet: 38614:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1227.866551] LNetError: 38614:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1227.901068] LNet: Removed LNI 192.168.202.133@tcp [ 1229.096846] Key type .llcrypt unregistered [ 1229.101287] Key type ._llcrypt unregistered [ 1256.034901] Key type ._llcrypt registered [ 1256.040282] Key type .llcrypt registered [ 1256.132745] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1256.995207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1257.083546] alg: No test for adler32 (adler32-zlib) [ 1258.181585] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 1258.373953] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 1260.039372] Key type lgssc registered [ 1261.652834] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1269.591618] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1269.605213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1271.080180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1275.845524] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1281.507387] LustreError: 40030:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1281.532515] LustreError: 40030:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1285.676977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1286.067687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1291.104990] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1300.352744] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1307.729508] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1316.614093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1316.790776] LustreError: 41855:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1317.255515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1322.487787] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1324.320187] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1333.919975] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1341.152719] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.33@tcp (stopping) [ 1342.434435] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1342.447267] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1342.474599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1346.020808] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1346.028345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1346.054742] Lustre: Skipped 1 previous similar message [ 1346.078544] Lustre: Skipped 2 previous similar messages [ 1348.001414] Lustre: server umount lustre-MDT0000 complete [ 1351.146929] LustreError: 40027:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1351.173279] LustreError: 40027:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1352.843509] LustreError: 40013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393502 with bad export cookie 3098121037810391666 [ 1352.844754] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1352.852079] LustreError: 40013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1356.256907] LustreError: 40026:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1356.257440] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1356.283653] LustreError: 40026:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1356.318886] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1356.327514] Lustre: Skipped 1 previous similar message [ 1359.544825] Lustre: server umount lustre-MDT0001 complete [ 1375.975913] Lustre: server umount lustre-OST0000 complete [ 1395.299681] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 1399.428371] Key type lgssc unregistered [ 1400.000841] LNet: 43895:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1400.018859] LNetError: 43895:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1400.051202] LNet: Removed LNI 192.168.202.133@tcp [ 1401.129418] Key type .llcrypt unregistered [ 1401.133453] Key type ._llcrypt unregistered [ 1430.637302] Key type ._llcrypt registered [ 1430.642257] Key type .llcrypt registered [ 1430.762859] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1431.904643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1431.930843] alg: No test for adler32 (adler32-zlib) [ 1433.157486] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 1433.432950] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 1435.143301] Key type lgssc registered [ 1436.319160] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1444.965840] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1444.988755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1446.491811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1451.757818] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1456.615732] LustreError: 45314:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1456.652894] LustreError: 45314:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1461.342090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1461.828366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1467.121829] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1477.092968] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1484.037564] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1491.624808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1492.119677] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1493.215051] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1499.003067] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1507.082938] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1515.264537] Lustre: lustre-OST0000: Not available for connect from 192.168.202.33@tcp (stopping) [ 1518.059784] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1518.061966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1518.081566] Lustre: Skipped 1 previous similar message [ 1520.323214] Lustre: lustre-OST0000: Not available for connect from 192.168.202.33@tcp (stopping) [ 1520.331032] Lustre: Skipped 1 previous similar message [ 1523.194559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1523.209258] Lustre: Skipped 1 previous similar message [ 1528.290227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1528.302834] Lustre: Skipped 2 previous similar messages [ 1529.311185] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1529.506467] Lustre: server umount lustre-OST0000 complete [ 1553.890657] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1553.894724] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1553.898968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1556.029784] Lustre: server umount lustre-MDT0000 complete [ 1557.984805] LustreError: 45315:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1558.012094] LustreError: 45315:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1560.410772] LustreError: 45296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393710 with bad export cookie 16845119675853205312 [ 1560.414662] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1560.421721] LustreError: 45296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1567.001239] Lustre: server umount lustre-MDT0001 complete [ 1580.172368] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 1583.182987] Key type lgssc unregistered [ 1583.451694] LNet: 49027:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1583.458845] LNetError: 49027:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1584.485346] LNet: Removed LNI 192.168.202.133@tcp [ 1585.493797] Key type .llcrypt unregistered [ 1585.496406] Key type ._llcrypt unregistered [ 1613.739956] Key type ._llcrypt registered [ 1613.741933] Key type .llcrypt registered [ 1613.827181] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1615.095410] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1615.140798] alg: No test for adler32 (adler32-zlib) [ 1616.275835] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 1616.517660] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 1618.151386] Key type lgssc registered [ 1619.305603] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1627.481356] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1627.514717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1628.864157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1634.825828] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1638.399789] LustreError: 50447:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1638.457527] LustreError: 50447:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1643.514269] LustreError: 50448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1648.537095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1648.612225] LustreError: 50447:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1648.945544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1654.282986] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1663.795783] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1672.646057] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1680.259668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1680.644205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1686.642716] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1693.660448] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1698.106254] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1703.866311] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1707.588524] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1719.675835] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-22 6h17m47s [ 1721.475237] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1723.875427] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1723.879508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1723.893768] Lustre: Skipped 1 previous similar message [ 1723.911161] Lustre: Skipped 2 previous similar messages [ 1724.963981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1724.982564] Lustre: Skipped 2 previous similar messages [ 1726.219815] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.33@tcp (stopping) [ 1726.240106] Lustre: Skipped 1 previous similar message [ 1729.934199] Lustre: server umount lustre-MDT0000 complete [ 1730.239395] LustreError: 52571:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1733.377233] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1736.383249] LustreError: 50444:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1736.405184] LustreError: 50444:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1740.658208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1740.847525] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1741.194327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1741.284610] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1742.190620] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1745.901240] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1749.385349] Lustre: DEBUG MARKER: Wait for df (36575) ... [ 1751.163949] Lustre: DEBUG MARKER: done [ 1755.619434] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1755.631227] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1755.657336] Lustre: Skipped 1 previous similar message [ 1755.666964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1756.640361] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1756.648450] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1761.852798] Lustre: server umount lustre-OST0000 complete [ 1767.905052] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1767.912699] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1767.933489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1767.933499] Lustre: Skipped 3 previous similar messages [ 1772.511041] Lustre: server umount lustre-MDT0000 complete [ 1775.073586] LustreError: 50448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1775.088810] LustreError: 50448:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1776.286758] LustreError: 50428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787393926 with bad export cookie 5821998896341672160 [ 1776.296084] LustreError: 50428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1776.297436] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1776.535331] Lustre: server umount lustre-MDT0001 complete [ 1788.234587] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 1791.070297] Key type lgssc unregistered [ 1791.374032] LNet: 55372:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1791.386316] LNetError: 55372:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1791.400209] LNet: Removed LNI 192.168.202.133@tcp [ 1792.214269] Key type .llcrypt unregistered [ 1792.220194] Key type ._llcrypt unregistered [ 1834.306225] Key type ._llcrypt registered [ 1834.308463] Key type .llcrypt registered [ 1834.401928] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 1835.389996] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1835.434139] alg: No test for adler32 (adler32-zlib) [ 1836.491090] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 1836.715615] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 1838.375165] Key type lgssc registered [ 1839.455846] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1847.405983] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1847.424915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1848.784735] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1848.818408] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1849.058172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1854.039837] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1862.482825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1862.638836] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1862.692914] Lustre: 57581:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1863.027048] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1868.637361] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1877.553507] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1885.207412] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1893.504418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1893.722544] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1893.983487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1900.654665] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 1903.248831] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1908.852397] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1915.116148] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-22 6h21m03s [ 1916.877695] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1924.112822] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1926.728541] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1926.821635] LustreError: 59700:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1947.376593] Lustre: lustre-MDT0000: Client e40ad33c-d6fe-4f32-83e0-bcbfe46d8eba (at 192.168.202.33@tcp) reconnecting [ 1966.872095] LustreError: 59700:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1970.733895] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1974.203100] Lustre: DEBUG MARKER: done [ 1980.390364] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1980.396159] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1980.401405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1980.403237] Lustre: Skipped 1 previous similar message [ 1985.511749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1985.515292] Lustre: Skipped 2 previous similar messages [ 1986.366267] Lustre: server umount lustre-OST0000 complete [ 1990.624841] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1990.641722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1990.656161] Lustre: Skipped 1 previous similar message [ 1990.664650] Lustre: Skipped 1 previous similar message [ 1996.567503] Lustre: server umount lustre-MDT0000 complete [ 2000.040778] LustreError: 57572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394150 with bad export cookie 3493415787829452068 [ 2000.056340] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2000.059894] LustreError: 57572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2000.424845] Lustre: server umount lustre-MDT0001 complete [ 2012.539573] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 2015.453737] Key type lgssc unregistered [ 2015.835709] LNet: 61914:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2015.841170] LNetError: 61914:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2015.857866] LNet: Removed LNI 192.168.202.133@tcp [ 2016.907112] Key type .llcrypt unregistered [ 2016.912649] Key type ._llcrypt unregistered [ 2066.382833] Key type ._llcrypt registered [ 2066.384987] Key type .llcrypt registered [ 2066.549611] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2068.411880] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2068.472071] alg: No test for adler32 (adler32-zlib) [ 2069.646326] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 2070.005129] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 2071.743557] Key type lgssc registered [ 2073.369937] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2078.637754] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2078.647806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2079.943101] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2079.971427] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2080.269579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2081.501981] Lustre: Failing over lustre-MDT0000 [ 2082.229681] Lustre: server umount lustre-MDT0000 complete [ 2090.381672] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 06:23:58 (1787394238) [ 2098.688603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2099.040085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2104.549179] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2113.153291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2113.334246] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2113.389807] Lustre: 64722:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2113.893502] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2119.304483] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2128.976059] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2136.845153] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2143.630318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2144.059084] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2144.348755] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2151.738915] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2153.591761] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2160.049986] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2166.309903] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2174.435450] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2174.453143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2174.456130] Lustre: Skipped 1 previous similar message [ 2174.477629] Lustre: Skipped 2 previous similar messages [ 2179.560217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2179.574258] Lustre: Skipped 2 previous similar messages [ 2180.948375] Lustre: server umount lustre-MDT0000 complete [ 2185.741103] LustreError: 64715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394335 with bad export cookie 2362641926669294944 [ 2185.745978] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2185.753969] LustreError: 64715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2186.098905] Lustre: server umount lustre-MDT0001 complete [ 2190.473809] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2196.706516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2205.151346] Lustre: 63645:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787394339/real 1787394339] req@00000000891d187f x1874218681673984/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787394355 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2205.188042] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2205.210820] Lustre: Skipped 1 previous similar message [ 2206.249825] Lustre: 63645:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787394340/real 1787394340] req@00000000b148d6ec x1874218681674112/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787394356 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2206.422961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2210.271353] Lustre: 63647:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787394344/real 1787394344] req@0000000012d4db18 x1874218681674240/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787394360 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2210.469765] LustreError: 63643:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a4446ccd x1874218681674880/t0(0) o250->MGC192.168.202.133@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2210.754604] LustreError: 66570:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2210.765276] LustreError: 66570:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2210.816585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2210.909939] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2214.959450] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2214.961326] LustreError: 66571:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2215.983067] LustreError: 68492:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2216.013546] LustreError: 68492:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2216.259293] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2216.991145] Lustre: 63645:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787394351/real 1787394351] req@00000000220a6f6c x1874218681674624/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787394367 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2217.025663] Lustre: 63645:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2221.032577] LustreError: 66570:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2221.049357] LustreError: 66570:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2225.510271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2225.967473] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2230.719683] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2231.286320] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2240.170159] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2246.882820] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2256.875174] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2256.875362] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2256.893347] Lustre: Skipped 2 previous similar messages [ 2256.893774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2261.087227] Lustre: server umount lustre-MDT0000 complete [ 2262.000804] LustreError: 66571:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2262.031245] LustreError: 66571:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2266.021816] LustreError: 64713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394416 with bad export cookie 2362641926669297219 [ 2266.026417] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2266.039508] LustreError: 64713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2266.570458] Lustre: server umount lustre-MDT0001 complete [ 2273.761376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2283.488839] Lustre: 63647:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787394417/real 1787394417] req@00000000b9f3e894 x1874218681701248/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787394433 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2283.528863] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2284.347526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2291.744358] LustreError: 63643:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000e7f04252 x1874218681702272/t0(0) o250->MGC192.168.202.133@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2292.160239] LustreError: 66571:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2292.203095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2292.244114] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2292.687161] Lustre: 63647:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787394426/real 1787394426] req@000000000e62f56d x1874218681701760/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787394442 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2292.730793] Lustre: 63647:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 2297.240657] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2305.692746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2311.149418] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2311.289468] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2319.737219] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2325.962609] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2336.742488] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2336.746938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2336.759656] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2336.762525] Lustre: Skipped 2 previous similar messages [ 2339.750382] Lustre: server umount lustre-MDT0000 complete [ 2342.893163] LustreError: 71176:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2342.916651] LustreError: 71176:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 2343.884666] LustreError: 64713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394493 with bad export cookie 2362641926669298598 [ 2343.890255] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2343.894319] LustreError: 64713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2344.214459] Lustre: server umount lustre-MDT0001 complete [ 2351.073148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2355.244789] Lustre: DEBUG MARKER: files compared the same [ 2368.187600] Lustre: server umount lustre-OST0000 complete [ 2387.148561] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 2390.113677] Key type lgssc unregistered [ 2390.414558] LNet: 74454:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2390.421065] LNetError: 74454:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2391.462600] LNet: Removed LNI 192.168.202.133@tcp [ 2392.538114] Key type .llcrypt unregistered [ 2392.539807] Key type ._llcrypt unregistered [ 2419.207896] Key type ._llcrypt registered [ 2419.218248] Key type .llcrypt registered [ 2419.383159] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2420.568506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2420.593121] alg: No test for adler32 (adler32-zlib) [ 2421.970151] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 2422.440881] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 2424.167201] Key type lgssc registered [ 2426.286410] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2435.636813] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2435.661578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2437.183505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2442.335058] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug +malloc all [ 2447.331695] LustreError: 75887:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2447.340896] LustreError: 75887:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2452.022652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2452.341295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2459.083289] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug +malloc all [ 2467.473608] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2474.274531] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2482.005968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2482.156259] LustreError: 77708:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2482.457392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2487.806480] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2489.366560] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug +malloc all [ 2498.977300] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2512.351692] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2512.358269] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2512.367783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2512.369997] Lustre: Skipped 1 previous similar message [ 2512.392122] Lustre: Skipped 1 previous similar message [ 2514.489755] Lustre: server umount lustre-OST0000 complete [ 2522.592248] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2522.596023] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2522.625952] Lustre: Skipped 1 previous similar message [ 2522.642390] Lustre: Skipped 1 previous similar message [ 2525.467401] Lustre: server umount lustre-MDT0000 complete [ 2527.716403] LustreError: 75882:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2527.744263] LustreError: 75882:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2529.414544] LustreError: 75868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394679 with bad export cookie 9342060842080761429 [ 2529.428277] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2529.670744] Lustre: server umount lustre-MDT0001 complete [ 2541.735870] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 2544.514805] Key type lgssc unregistered [ 2544.839794] LNet: 79420:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2544.846674] LNetError: 79420:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2545.893440] LNet: Removed LNI 192.168.202.133@tcp [ 2546.858387] Key type .llcrypt unregistered [ 2546.860087] Key type ._llcrypt unregistered [ 2579.449430] Key type ._llcrypt registered [ 2579.452409] Key type .llcrypt registered [ 2579.606583] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2581.004329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2581.281674] alg: No test for adler32 (adler32-zlib) [ 2582.428657] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 2582.643281] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 2584.343278] Key type lgssc registered [ 2585.438434] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2593.029674] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2593.058197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2608.735610] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2613.855328] LustreError: 80820:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.133@tcp: failed processing log, type 4: rc = -110 [ 2640.479201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2647.583813] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2656.145532] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2664.847710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2665.051495] Lustre: *** cfs_fail_loc=706, val=0*** [ 2665.634302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2665.734346] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2670.022777] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2671.073717] LustreError: 80846:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2671.103414] LustreError: 80846:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2675.190114] LustreError: 81906:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2675.246172] LustreError: 81906:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2678.029765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2678.414904] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2681.932639] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2688.854753] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2694.546437] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2698.722090] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2698.727654] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2698.753469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2699.236538] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2699.259394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2699.266131] Lustre: Skipped 1 previous similar message [ 2704.357569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2709.476032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2709.479030] Lustre: Skipped 1 previous similar message [ 2711.007451] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2711.191847] Lustre: server umount lustre-OST0000 complete [ 2719.201672] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2719.206600] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2719.216197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2721.736834] Lustre: server umount lustre-MDT0000 complete [ 2724.832564] LustreError: 81905:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2724.845047] LustreError: 81905:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2725.828983] LustreError: 80827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787394875 with bad export cookie 34955026855120619 [ 2725.833662] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2725.854479] LustreError: 80827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2726.434988] Lustre: server umount lustre-MDT0001 complete [ 2739.200624] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 2742.043725] Key type lgssc unregistered [ 2742.343775] LNet: 84436:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2742.355823] LNetError: 84436:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2743.405051] LNet: Removed LNI 192.168.202.133@tcp [ 2744.472103] Key type .llcrypt unregistered [ 2744.474993] Key type ._llcrypt unregistered [ 2772.384743] Key type ._llcrypt registered [ 2772.387383] Key type .llcrypt registered [ 2772.497507] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2773.653732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2773.671934] alg: No test for adler32 (adler32-zlib) [ 2774.859385] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 2775.175325] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 2776.856172] Key type lgssc registered [ 2778.440296] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2788.200822] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2788.229229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2794.981663] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2804.341527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2805.514829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2810.390314] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2810.848567] LustreError: 86542:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2810.874183] LustreError: 86542:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2815.968894] LustreError: 86546:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2815.994943] LustreError: 86546:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2819.942713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2820.379442] LustreError: 86542:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2820.420481] LustreError: 86542:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2820.438894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2825.703211] LustreError: 87291:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2825.723328] LustreError: 87291:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2827.045712] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2830.821119] LustreError: 86542:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2830.845073] LustreError: 86542:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2836.525448] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2836.554474] Lustre: MGS: Client 0c5a3613-c9fd-4efa-9744-17b81064ae06 (at 0@lo) reconnecting [ 2836.637121] LustreError: 86556:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2836.971359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2837.124410] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2842.712801] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 2858.464669] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2858.475994] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2858.493749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2862.566170] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2862.571139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2862.574337] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2862.617923] Lustre: Skipped 1 previous similar message [ 2864.060842] Lustre: server umount lustre-OST0000 complete [ 2872.800261] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2872.802512] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2872.812921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2872.831782] Lustre: Skipped 1 previous similar message [ 2872.844234] Lustre: Skipped 1 previous similar message [ 2874.123503] Lustre: server umount lustre-MDT0000 complete [ 2877.923342] LustreError: 88302:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2877.949444] LustreError: 88302:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2878.807579] Lustre: server umount lustre-MDT0001 complete [ 2882.304643] Lustre: server umount MGS complete [ 2895.500950] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 06:37:24 (1787395044) [ 2931.286200] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 2943.652145] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 2985.362972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2990.586294] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3000.254080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3000.625610] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3000.665659] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3000.743743] Lustre: lustre-MDT0001: new disk, initializing [ 3000.827290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3000.861759] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3000.872067] Lustre: 93729:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3000.880174] LustreError: 93729:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3005.502186] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3016.929706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3017.191838] Lustre: lustre-OST0000: new disk, initializing [ 3017.196690] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3017.203852] Lustre: 94496:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3017.254156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3024.546088] Lustre: 94752:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000035eb7c50 x1874219420919296/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787395190 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3026.079412] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3037.764383] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3044.833731] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3046.905617] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3046.919559] Lustre: MGS: Client d122288d-3127-4589-81da-a2f4660fe28c (at 0@lo) reconnecting [ 3047.015810] Lustre: 95542:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3047.046811] LustreError: 93715:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3047.117943] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3047.260617] Lustre: lustre-MDT0000: new disk, initializing [ 3047.411724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3047.447618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3048.482752] Lustre: 95566:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000c28e1b8a x1874219420926848/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787395214 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3053.208839] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3053.538389] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3057.581072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3057.597479] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3057.687557] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3080.674265] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3080.696450] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3080.727956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3085.873866] Lustre: server umount lustre-OST0000 complete [ 3091.425911] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3091.433205] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3091.441185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3091.441193] Lustre: Skipped 2 previous similar messages [ 3091.445748] Lustre: Skipped 2 previous similar messages [ 3101.670469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3101.673483] Lustre: Skipped 4 previous similar messages [ 3105.247266] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3105.436788] Lustre: server umount lustre-MDT0000 complete [ 3106.787752] LustreError: 95959:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3106.830738] LustreError: 95959:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3116.678333] Lustre: server umount lustre-MDT0001 complete [ 3121.616512] Lustre: server umount MGS complete [ 3135.497716] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 06:41:23 (1787395283) [ 3158.269223] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 3162.213102] Key type lgssc unregistered [ 3162.798487] LNet: 98064:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3162.810861] LNetError: 98064:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3162.849051] LNet: Removed LNI 192.168.202.133@tcp [ 3164.109554] Key type .llcrypt unregistered [ 3164.114197] Key type ._llcrypt unregistered [ 3186.686752] Key type ._llcrypt registered [ 3186.689880] Key type .llcrypt registered [ 3186.791487] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 3187.811273] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3187.969081] alg: No test for adler32 (adler32-zlib) [ 3189.230128] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 3189.563835] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 3191.224101] Key type lgssc registered [ 3192.428787] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3200.942724] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3200.962894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3202.463534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3207.356757] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3212.772851] LustreError: 99288:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3212.784879] LustreError: 99288:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3216.166761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3216.802752] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3221.457508] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3229.220197] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3235.565843] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3242.622356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3243.013520] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3247.098343] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3250.419206] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3258.908985] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3271.649935] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3271.650799] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3271.658120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3271.659257] Lustre: Skipped 1 previous similar message [ 3276.160636] Lustre: server umount lustre-OST0000 complete [ 3281.889285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3281.899072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3281.903144] Lustre: Skipped 1 previous similar message [ 3287.010336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3287.018057] Lustre: Skipped 1 previous similar message [ 3287.842097] Lustre: server umount lustre-MDT0000 complete [ 3291.107711] LustreError: 100295:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3291.156550] LustreError: 100295:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3292.463732] LustreError: 99268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395442 with bad export cookie 15369273948567678018 [ 3292.466849] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3292.474526] LustreError: 99268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3292.788466] Lustre: server umount lustre-MDT0001 complete [ 3305.735530] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 3308.592080] Key type lgssc unregistered [ 3308.959229] LNet: 102824:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3308.970255] LNetError: 102824:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3308.988666] LNet: Removed LNI 192.168.202.133@tcp [ 3309.914667] Key type .llcrypt unregistered [ 3309.917064] Key type ._llcrypt unregistered [ 3328.081946] Key type ._llcrypt registered [ 3328.086797] Key type .llcrypt registered [ 3328.221138] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 3329.320539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3329.353818] alg: No test for adler32 (adler32-zlib) [ 3330.691711] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 3330.902427] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 3332.559472] Key type lgssc registered [ 3333.876748] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3340.554756] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3340.565171] LustreError: 103903:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3340.616797] LustreError: 103910:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3340.632098] LustreError: 103903:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4943 [ 3340.636687] LustreError: 103903:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3340.645786] LustreError: 103903:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3340.659569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3342.314820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3350.460648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3350.838061] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3355.617830] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3362.077804] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing lsmod [ 3365.441782] LustreError: 105115:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3365.446696] LustreError: 105116:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3365.457973] LustreError: 105115:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4995 [ 3365.468164] LustreError: 105115:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3365.472832] LustreError: 105115:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3365.479528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3365.882369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3367.914920] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3371.490501] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3371.506830] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3371.528479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3373.026075] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3373.029567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3373.046845] Lustre: Skipped 1 previous similar message [ 3373.059716] Lustre: Skipped 2 previous similar messages [ 3375.018108] Lustre: server umount lustre-MDT0000 complete [ 3375.142203] LustreError: 103947:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3375.169632] LustreError: 103947:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3378.725963] LustreError: 103925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395528 with bad export cookie 13767141020444064146 [ 3378.740667] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3379.322976] LustreError: 104378:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000f25bf2de x1874220002857600/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3379.467738] Lustre: server umount lustre-MDT0001 complete [ 3393.196943] Lustre: server umount lustre-OST0000 complete [ 3402.080682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3402.539677] LustreError: 106178:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3402.629890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3407.016674] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3407.853242] LustreError: 106179:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3412.965851] LustreError: 106178:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3416.055894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3416.343584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3421.296669] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3429.325164] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3435.111238] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3442.526409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3442.931512] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3447.027106] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3449.561215] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3457.475818] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3470.827314] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3470.835121] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3470.855213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3471.849129] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3476.944495] Lustre: server umount lustre-OST0000 complete [ 3482.081756] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3482.098939] Lustre: Skipped 1 previous similar message [ 3482.108510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3482.110854] Lustre: Skipped 2 previous similar messages [ 3487.200355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3487.213090] Lustre: Skipped 1 previous similar message [ 3487.829075] Lustre: server umount lustre-MDT0000 complete [ 3491.300073] LustreError: 106178:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3491.318694] LustreError: 106178:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3491.834483] LustreError: 106160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395641 with bad export cookie 13767141020444065063 [ 3491.848677] LustreError: 106160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3491.855173] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3492.129082] Lustre: server umount lustre-MDT0001 complete [ 3504.117784] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 3506.964776] Key type lgssc unregistered [ 3507.238692] LNet: 109730:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3507.241920] LNetError: 109730:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3508.262579] LNet: Removed LNI 192.168.202.133@tcp [ 3509.313382] Key type .llcrypt unregistered [ 3509.319272] Key type ._llcrypt unregistered [ 3538.780540] Key type ._llcrypt registered [ 3538.796947] Key type .llcrypt registered [ 3539.061324] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 3540.855354] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3541.251863] alg: No test for adler32 (adler32-zlib) [ 3542.752389] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 3543.323861] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 3545.087284] Key type lgssc registered [ 3546.565955] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3554.948764] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3554.968338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3556.323276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3560.666341] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3566.562499] LustreError: 111155:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3566.576316] LustreError: 111155:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3568.824258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3569.160563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3573.455400] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3581.026643] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3586.422945] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3592.772599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3593.002165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3598.064320] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3601.225942] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3606.528176] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3611.688299] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3616.089905] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3621.857750] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3621.877280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3621.880432] Lustre: Skipped 1 previous similar message [ 3621.903745] Lustre: Skipped 1 previous similar message [ 3627.084550] Lustre: server umount lustre-OST0000 complete [ 3632.099437] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3632.102437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3632.124423] Lustre: Skipped 1 previous similar message [ 3632.147970] Lustre: Skipped 1 previous similar message [ 3637.217979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3637.229553] Lustre: Skipped 1 previous similar message [ 3642.339835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3645.409051] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3645.587768] Lustre: server umount lustre-MDT0000 complete [ 3647.458768] LustreError: 111155:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3647.484724] LustreError: 111155:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3649.832183] LustreError: 111136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395799 with bad export cookie 5232189406022123064 [ 3649.836258] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3649.839728] LustreError: 111136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3650.157823] Lustre: server umount lustre-MDT0001 complete [ 3661.875636] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 3664.606411] Key type lgssc unregistered [ 3664.958245] LNet: 114793:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3664.973352] LNetError: 114793:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3666.033397] LNet: Removed LNI 192.168.202.133@tcp [ 3666.871154] Key type .llcrypt unregistered [ 3666.874325] Key type ._llcrypt unregistered [ 3687.042432] Key type ._llcrypt registered [ 3687.046860] Key type .llcrypt registered [ 3687.172729] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 3688.856404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3688.911085] alg: No test for adler32 (adler32-zlib) [ 3690.113691] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 3690.296465] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 3692.015930] Key type lgssc registered [ 3693.077800] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3701.719170] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3701.736885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3703.201612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3708.219520] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3713.509215] LustreError: 116016:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3713.524791] LustreError: 116016:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3717.882945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3718.241318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3723.175709] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3732.119908] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3738.184815] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3747.271703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3747.758170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3748.799715] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3755.553699] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3763.981650] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3777.857201] Lustre: server umount lustre-OST0000 complete [ 3783.651189] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3783.665453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3783.668210] Lustre: Skipped 1 previous similar message [ 3788.770744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3788.777142] Lustre: Skipped 1 previous similar message [ 3792.965181] Lustre: server umount lustre-MDT0000 complete [ 3793.889756] LustreError: 116017:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3793.920156] LustreError: 116017:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3794.912273] Lustre: 115330:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787395928/real 1787395928] req@000000007f1f4526 x1874220380352384/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787395944 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3794.960221] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3794.977267] Lustre: Skipped 1 previous similar message [ 3794.991884] LustreError: 116016:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3795.005861] LustreError: 116016:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3797.831024] LustreError: 115996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787395947 with bad export cookie 1801374899279902417 [ 3797.842063] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3798.191953] Lustre: server umount lustre-MDT0001 complete [ 3812.086102] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 3815.962394] Key type lgssc unregistered [ 3816.402871] LNet: 119551:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3816.416042] LNetError: 119551:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3817.446812] LNet: Removed LNI 192.168.202.133@tcp [ 3818.442905] Key type .llcrypt unregistered [ 3818.447944] Key type ._llcrypt unregistered [ 3845.830149] Key type ._llcrypt registered [ 3845.832388] Key type .llcrypt registered [ 3845.992752] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 3847.640357] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3847.695303] alg: No test for adler32 (adler32-zlib) [ 3848.756258] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 3848.966387] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 3850.679254] Key type lgssc registered [ 3851.996325] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3861.786775] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3861.824834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3863.439417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3869.373132] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3873.761699] LustreError: 120971:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3873.787353] LustreError: 120971:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3878.890498] LustreError: 120972:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3880.291290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3880.509153] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3885.502376] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3894.803209] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3901.915775] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3910.964044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3911.381108] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3918.669593] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 3928.070572] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3933.951628] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3946.245041] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3951.039271] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3951.043806] Lustre: Skipped 2 previous similar messages [ 3955.651776] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3955.660993] Lustre: Skipped 2 previous similar messages [ 3956.328391] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3956.356584] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3969.207491] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.33@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3969.217559] Lustre: Skipped 2 previous similar messages [ 3977.915332] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.33@tcp 192.168.0.[2,10]@tcp [ 3983.029123] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.33/24@tcp in log lustre-MDT0000 [ 3983.040411] Lustre: Skipped 2 previous similar messages [ 3986.928437] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.33/24@tcp [ 3986.935219] Lustre: Skipped 1 previous similar message [ 3995.626378] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3995.630107] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3995.656761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3995.675760] Lustre: Skipped 1 previous similar message [ 3995.700784] Lustre: Skipped 1 previous similar message [ 4000.739916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4000.749778] Lustre: Skipped 1 previous similar message [ 4001.425370] Lustre: server umount lustre-OST0000 complete [ 4005.856271] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4005.858834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4005.883820] Lustre: Skipped 1 previous similar message [ 4005.893883] Lustre: Skipped 1 previous similar message [ 4010.982508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4010.988601] Lustre: Skipped 1 previous similar message [ 4011.865126] Lustre: server umount lustre-MDT0000 complete [ 4015.018868] LustreError: 120953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396165 with bad export cookie 7384685015502635050 [ 4015.021464] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4015.031425] LustreError: 120953:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4015.371545] Lustre: server umount lustre-MDT0001 complete [ 4027.746100] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 4031.521549] Key type lgssc unregistered [ 4032.064566] LNet: 125833:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4032.073281] LNetError: 125833:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4032.097910] LNet: Removed LNI 192.168.202.133@tcp [ 4033.101803] Key type .llcrypt unregistered [ 4033.103451] Key type ._llcrypt unregistered [ 4066.973208] Key type ._llcrypt registered [ 4066.978915] Key type .llcrypt registered [ 4067.075532] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 4068.674619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4068.715074] alg: No test for adler32 (adler32-zlib) [ 4069.881281] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 4070.253189] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 4071.943164] Key type lgssc registered [ 4073.486989] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4081.457634] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4081.490406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4082.925663] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4082.950049] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.33/24@tcp [ 4082.974412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4087.770920] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4092.385575] LustreError: 127396:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4092.402308] LustreError: 127396:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4095.961839] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4096.342219] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4096.381549] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.33/24@tcp [ 4096.428295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4101.063955] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4109.346494] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4116.538339] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4125.136893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4125.585436] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4127.670395] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 4132.439935] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4143.340830] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4157.409294] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4157.410944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4157.441273] Lustre: Skipped 1 previous similar message [ 4157.458217] Lustre: Skipped 1 previous similar message [ 4157.726407] Lustre: server umount lustre-OST0000 complete [ 4162.532120] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4162.534763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4162.545135] Lustre: Skipped 1 previous similar message [ 4162.569491] Lustre: Skipped 1 previous similar message [ 4167.649632] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4167.656530] Lustre: Skipped 1 previous similar message [ 4172.769570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4172.778168] Lustre: Skipped 1 previous similar message [ 4176.351143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4176.609670] Lustre: server umount lustre-MDT0000 complete [ 4177.893057] LustreError: 128406:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4177.941733] LustreError: 128406:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4181.014201] LustreError: 127376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396331 with bad export cookie 7435975379751207213 [ 4181.026202] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4181.468781] Lustre: server umount lustre-MDT0001 complete [ 4193.226400] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 4196.237749] Key type lgssc unregistered [ 4196.602584] LNet: 130986:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4196.624354] LNetError: 130986:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4196.660991] LNet: Removed LNI 192.168.202.133@tcp [ 4197.577203] Key type .llcrypt unregistered [ 4197.580716] Key type ._llcrypt unregistered [ 4226.673299] Key type ._llcrypt registered [ 4226.675068] Key type .llcrypt registered [ 4226.880470] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 4246.962534] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 4248.354894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4248.443379] alg: No test for adler32 (adler32-zlib) [ 4249.549031] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 4249.816552] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 4251.575155] Key type lgssc registered [ 4252.883284] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4295.779546] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4295.821895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4297.358899] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4297.457874] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4297.600474] Lustre: lustre-MDT0000: new disk, initializing [ 4297.690857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4297.712600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4303.514629] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4317.642738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4317.858937] Lustre: 134392:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4317.997561] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4318.008944] Lustre: Skipped 1 previous similar message [ 4318.223352] Lustre: lustre-MDT0001: new disk, initializing [ 4318.386175] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4318.450122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4318.465207] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4323.578669] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4334.897215] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4342.433280] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4350.364954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4350.738547] Lustre: lustre-OST0000: new disk, initializing [ 4350.744765] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4350.751992] Lustre: 136372:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4350.865476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4358.285839] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4360.820702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4360.837493] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4360.977106] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4370.084950] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4376.146677] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4376.448247] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4381.685318] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4381.947042] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4386.791152] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4386.794033] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4386.803176] LustreError: Skipped 1 previous similar message [ 4386.804912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4386.836858] Lustre: Skipped 1 previous similar message [ 4390.916708] Lustre: server umount lustre-OST0000 complete [ 4397.028255] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4397.060383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4397.068043] Lustre: Skipped 2 previous similar messages [ 4402.146420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4402.154479] Lustre: Skipped 1 previous similar message [ 4406.243130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4406.247824] Lustre: Skipped 1 previous similar message [ 4410.335215] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4410.513300] Lustre: server umount lustre-MDT0000 complete [ 4414.831572] LustreError: 136371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396564 with bad export cookie 18352007153324263208 [ 4414.860334] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4414.866719] LustreError: 136371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4415.665574] Lustre: server umount lustre-MDT0001 complete [ 4425.196384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4425.832318] LustreError: 138588:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4425.858694] LustreError: 138588:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4425.941471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4431.322612] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4431.329300] LustreError: 138589:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4435.426485] LustreError: 138588:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4440.549543] LustreError: 138589:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4441.221611] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4441.533341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4446.368887] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4455.140935] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4461.697876] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4471.174990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4471.564467] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4479.767812] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4489.596163] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4494.897798] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4496.248497] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 4500.395211] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4500.612961] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4510.337683] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4510.569553] Lustre: lustre-OST0001: new disk, initializing [ 4510.582265] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4510.600244] Lustre: 141967:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4510.757683] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4516.517777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4516.528248] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4516.695877] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4518.555850] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4528.634290] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4528.862223] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4534.504554] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4534.762881] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4543.251384] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d578506e800.ost_server_uuid 50 [ 4545.380257] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8d578506e800.ost_server_uuid in IDLE state after 0 sec [ 4557.799816] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4557.806865] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4557.807163] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4557.818838] Lustre: Skipped 2 previous similar messages [ 4557.847479] Lustre: Skipped 1 previous similar message [ 4561.931675] Lustre: server umount lustre-OST0001 complete [ 4562.919000] LustreError: 140408:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4562.965257] LustreError: 140408:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4567.011099] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4567.020917] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4567.048691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4572.677522] Lustre: server umount lustre-OST0000 complete [ 4578.276285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4578.321876] Lustre: Skipped 2 previous similar messages [ 4583.395793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4583.405113] Lustre: Skipped 5 previous similar messages [ 4583.697964] Lustre: server umount lustre-MDT0000 complete [ 4588.385012] LustreError: 138569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787396738 with bad export cookie 18352007153324264090 [ 4588.389078] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4588.398990] LustreError: 138569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4588.776799] Lustre: server umount lustre-MDT0001 complete [ 4613.934517] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 4617.060053] Key type lgssc unregistered [ 4617.369904] LNet: 144976:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4617.381418] LNetError: 144976:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4617.395845] LNet: Removed LNI 192.168.202.133@tcp [ 4618.385524] Key type .llcrypt unregistered [ 4618.388600] Key type ._llcrypt unregistered [ 4666.729428] Key type ._llcrypt registered [ 4666.732437] Key type .llcrypt registered [ 4666.831963] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 4683.822902] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 4685.784610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4685.805950] alg: No test for adler32 (adler32-zlib) [ 4686.932215] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 4687.133791] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 4688.791283] Key type lgssc registered [ 4689.732365] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4730.418247] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4730.449941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4731.659598] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4731.680966] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4731.739421] Lustre: lustre-MDT0000: new disk, initializing [ 4731.793107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4731.804508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4736.607487] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4748.283912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4748.515802] Lustre: 149134:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4748.602362] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4748.605908] Lustre: Skipped 1 previous similar message [ 4748.695672] Lustre: lustre-MDT0001: new disk, initializing [ 4748.858618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4748.914433] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4748.928881] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4754.269724] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4765.395722] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4773.205478] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4781.664921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4782.045936] Lustre: lustre-OST0000: new disk, initializing [ 4782.060109] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4782.068874] Lustre: 151105:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4782.183762] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4789.533775] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4791.375115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4791.393944] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4791.510098] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4800.628376] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4805.659444] Lustre: Failing over lustre-OST0000 [ 4805.847505] Lustre: server umount lustre-OST0000 complete [ 4823.455392] Lustre: 147359:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787396957/real 1787396957] req@000000003e109af9 x1874221425789056/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787396973 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4823.489588] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4824.547799] Lustre: 147356:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787396957/real 1787396957] req@00000000f319fa27 x1874221425789184/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1787396973 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4824.581452] Lustre: 147356:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4825.251944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4825.677091] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4825.701754] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4828.624117] Lustre: 147357:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787396962/real 1787396962] req@00000000856ee0a0 x1874221425790208/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787396978 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4828.682795] Lustre: 147357:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4832.241702] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4833.801107] Lustre: 147358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787396967/real 1787396967] req@0000000056de13c5 x1874221425791488/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787396983 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4833.831089] Lustre: 147358:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4833.850049] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4834.023827] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4834.026684] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4836.592110] Lustre: Failing over lustre-MDT0000 [ 4837.132698] Lustre: server umount lustre-MDT0000 complete [ 4838.688216] Lustre: 147358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787396972/real 1787396972] req@00000000f6c6ade0 x1874221425792512/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787396988 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4838.716025] Lustre: 147358:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4838.884296] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4838.910654] Lustre: Skipped 2 previous similar messages [ 4844.004839] LustreError: 150298:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4844.028823] LustreError: 150298:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 4849.120530] LustreError: 149147:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4849.136850] LustreError: 149147:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4854.251327] Lustre: 147358:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787396988/real 1787396988] req@00000000cb84a9a5 x1874221425801984/t0(0) o400->MGC192.168.202.133@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787397004 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4854.263119] LustreError: 149146:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4854.281244] Lustre: 147358:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4854.281341] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4854.335573] LustreError: 149146:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4857.524195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4859.376652] LustreError: 149147:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4859.404799] LustreError: 149147:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 4864.487621] LustreError: 147355:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a0cc084b x1874221425806848/t0(0) o250->MGC192.168.202.133@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4864.550481] LustreError: 149146:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4864.571380] LustreError: 149146:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4864.981061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4865.028458] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4865.540629] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4870.125736] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4870.133862] Lustre: Skipped 1 previous similar message [ 4870.141922] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4870.177540] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4870.182705] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4880.353458] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4880.368721] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4880.375314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4880.394861] Lustre: Skipped 2 previous similar messages [ 4885.327615] Lustre: server umount lustre-OST0000 complete [ 4890.592329] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4890.603849] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4890.610898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4890.625204] Lustre: Skipped 1 previous similar message [ 4890.636997] Lustre: Skipped 1 previous similar message [ 4895.600322] Lustre: server umount lustre-MDT0000 complete [ 4895.716481] LustreError: 149143:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4895.741705] LustreError: 149143:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4900.395285] LustreError: 149931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397050 with bad export cookie 15543142327438853628 [ 4900.398611] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4900.405038] LustreError: 149931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4900.696859] Lustre: server umount lustre-MDT0001 complete [ 4916.797763] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 4919.979440] Key type lgssc unregistered [ 4920.292678] LNet: 154764:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4920.299267] LNetError: 154764:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4920.310830] LNet: Removed LNI 192.168.202.133@tcp [ 4921.168928] Key type .llcrypt unregistered [ 4921.177691] Key type ._llcrypt unregistered [ 4961.576631] Key type ._llcrypt registered [ 4961.580883] Key type .llcrypt registered [ 4961.693192] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 4962.618383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4963.024962] alg: No test for adler32 (adler32-zlib) [ 4964.089256] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 4964.274877] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 4965.903257] Key type lgssc registered [ 4967.391271] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4976.925829] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4976.949242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4978.397203] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4983.722906] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 4988.900064] LustreError: 156286:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4988.927940] LustreError: 156286:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4994.019480] LustreError: 156285:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4998.657784] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4999.142898] LustreError: 156286:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5003.238071] LustreError: 156285:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5003.319309] LustreError: 156314:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5003.340540] LustreError: 156314:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5003.351489] LustreError: 156314:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 25, retries 0, failed: rc = -5 [ 5003.699687] Lustre: server umount lustre-MDT0000 complete [ 5021.798890] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 5034.383440] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 5074.514314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5074.734357] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5074.761298] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5074.849069] Lustre: lustre-MDT0000: new disk, initializing [ 5074.958685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5075.003307] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5079.462682] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5092.735933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5092.939775] Lustre: 159617:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5093.039283] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5093.049364] Lustre: Skipped 1 previous similar message [ 5093.166481] Lustre: lustre-MDT0001: new disk, initializing [ 5093.389099] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5093.433207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5093.450530] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5098.463337] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5110.427335] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5119.066092] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5126.563838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5126.973374] Lustre: lustre-OST0000: new disk, initializing [ 5126.983098] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5126.993731] Lustre: 161586:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5127.153858] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5134.405972] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5135.955363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5135.967339] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5136.141361] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5145.454784] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5150.332761] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5152.795890] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 07:15:00 (1787397300) [ 5421.024857] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5421.036689] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5421.049333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5423.072275] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5423.073739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5423.098500] Lustre: Skipped 2 previous similar messages [ 5425.251516] Lustre: server umount lustre-MDT0000 complete [ 5428.194638] LustreError: 159630:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5428.222074] LustreError: 159630:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5429.795060] LustreError: 159611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397579 with bad export cookie 13354201244366005667 [ 5429.795603] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5429.807994] LustreError: 159611:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5433.314515] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5433.324410] Lustre: Skipped 1 previous similar message [ 5433.331740] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5439.467574] LustreError: 160781:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5439.469727] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5439.508766] LustreError: 160781:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5439.533583] Lustre: Skipped 1 previous similar message [ 5444.065732] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5444.692623] Lustre: server umount lustre-MDT0001 complete [ 5457.126213] Lustre: server umount lustre-OST0000 complete [ 5472.867132] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 5484.722250] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 5529.183457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5529.699443] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5529.742302] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5529.866888] Lustre: lustre-MDT0000: new disk, initializing [ 5529.970848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5529.987587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5535.550464] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5548.869134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5549.087806] Lustre: 166256:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5549.173525] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5549.179370] Lustre: Skipped 1 previous similar message [ 5549.241178] Lustre: lustre-MDT0001: new disk, initializing [ 5549.333660] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5549.363730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5549.378400] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5555.473760] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5569.293516] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5579.133430] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5588.860147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5589.132552] Lustre: lustre-OST0000: new disk, initializing [ 5589.135315] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5589.142678] Lustre: 168225:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5589.236528] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5595.578859] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5598.773216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5598.785398] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5598.905925] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5605.916272] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5611.868135] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5612.144194] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5617.527678] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5617.807242] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5624.808390] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5624.825393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5624.834020] Lustre: Skipped 1 previous similar message [ 5626.592909] Lustre: server umount lustre-OST0000 complete [ 5631.458166] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5631.463355] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5635.041612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5635.055384] Lustre: Skipped 3 previous similar messages [ 5636.965647] Lustre: server umount lustre-MDT0000 complete [ 5640.162590] LustreError: 167392:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5640.186451] LustreError: 167392:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5642.075174] LustreError: 166248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397792 with bad export cookie 13354201244366164497 [ 5642.087102] LustreError: 166248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5642.094257] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5642.491730] Lustre: server umount lustre-MDT0001 complete [ 5654.554206] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 07:23:23 (1787397803) [ 5673.669491] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 5686.079622] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 5725.793138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5726.083699] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5726.103788] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5726.169205] Lustre: lustre-MDT0000: new disk, initializing [ 5726.225482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5726.253762] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5731.363414] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5742.977768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5743.153231] Lustre: 172982:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5743.515696] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5749.145148] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5761.197736] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5768.793330] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5777.871878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5778.177377] Lustre: lustre-OST0000: new disk, initializing [ 5778.182938] Lustre: Skipped 1 previous similar message [ 5778.187123] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5778.191972] Lustre: Skipped 2 previous similar messages [ 5778.198210] Lustre: 174963:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5785.403468] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5786.228814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5786.234930] Lustre: Skipped 1 previous similar message [ 5786.237958] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5786.360689] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5796.338845] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5807.072350] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5807.079395] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5807.089623] LustreError: Skipped 1 previous similar message [ 5807.101673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5807.110207] Lustre: Skipped 2 previous similar messages [ 5807.123024] Lustre: Skipped 1 previous similar message [ 5811.151821] Lustre: server umount lustre-OST0000 complete [ 5821.988082] Lustre: server umount lustre-MDT0000 complete [ 5822.437962] LustreError: 172995:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5822.466071] LustreError: 172995:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5826.189980] LustreError: 172976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787397976 with bad export cookie 13354201244366165379 [ 5826.194428] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5826.210509] LustreError: 172976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5826.699387] Lustre: server umount lustre-MDT0001 complete [ 5836.769500] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 07:26:24 (1787397984) [ 5856.930366] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 5869.942840] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 5916.398682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5916.604588] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5916.629538] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5916.692420] Lustre: lustre-MDT0000: new disk, initializing [ 5916.766383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5916.771306] Lustre: Skipped 2 previous similar messages [ 5916.782911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5922.577227] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5934.445636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5934.591700] Lustre: 179290:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5934.891213] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5940.287130] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5951.368744] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5958.130844] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5966.042149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5966.364704] Lustre: 181264:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5973.187752] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 5975.132092] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5975.249628] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5984.384955] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5996.010977] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5996.012748] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5996.020224] LustreError: Skipped 1 previous similar message [ 5996.020832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5996.020836] Lustre: Skipped 2 previous similar messages [ 5996.074221] Lustre: Skipped 3 previous similar messages [ 5999.285551] Lustre: server umount lustre-OST0000 complete [ 6009.804449] Lustre: server umount lustre-MDT0000 complete [ 6014.286884] LustreError: 179281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787398164 with bad export cookie 13354201244366166464 [ 6014.305437] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6014.772291] Lustre: server umount lustre-MDT0001 complete [ 6029.679959] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 6032.623750] Key type lgssc unregistered [ 6032.939918] LNet: 183121:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6032.944552] LNetError: 183121:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6032.956953] LNet: Removed LNI 192.168.202.133@tcp [ 6033.926039] Key type .llcrypt unregistered [ 6033.930315] Key type ._llcrypt unregistered [ 6059.614453] Key type ._llcrypt registered [ 6059.616646] Key type .llcrypt registered [ 6059.701307] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 6061.309059] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6061.331981] alg: No test for adler32 (adler32-zlib) [ 6062.375511] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 6062.664976] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 6064.335185] Key type lgssc registered [ 6065.498712] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6073.771861] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6073.794231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6075.073564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6081.303659] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6085.605948] LustreError: 184540:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6085.631812] LustreError: 184540:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6090.723380] LustreError: 184539:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6092.445328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6093.010558] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6097.595431] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6106.925809] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6112.818839] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6121.021757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6121.339355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6127.771496] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6136.631873] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6145.952313] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6168.554552] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6168.567317] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6168.599165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6171.619628] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6171.648884] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6171.667259] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6173.614091] Lustre: server umount lustre-OST0000 complete [ 6180.322168] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6180.333956] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6180.366805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6180.376988] Lustre: Skipped 1 previous similar message [ 6184.955313] Lustre: server umount lustre-MDT0000 complete [ 6186.994947] LustreError: 186380:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6187.027097] LustreError: 186380:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6188.963286] LustreError: 184519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787398338 with bad export cookie 4912683449725707119 [ 6188.967694] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6188.980729] LustreError: 184519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6189.524385] Lustre: server umount lustre-MDT0001 complete [ 6201.422229] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 6204.383465] Key type lgssc unregistered [ 6204.643751] LNet: 188097:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6204.648233] LNetError: 188097:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6204.663514] LNet: Removed LNI 192.168.202.133@tcp [ 6205.559689] Key type .llcrypt unregistered [ 6205.562053] Key type ._llcrypt unregistered [ 6235.810682] Key type ._llcrypt registered [ 6235.815036] Key type .llcrypt registered [ 6235.994624] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 6237.885440] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6237.955896] alg: No test for adler32 (adler32-zlib) [ 6239.120359] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 6239.479925] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 6241.143193] Key type lgssc registered [ 6242.246811] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6250.480146] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6250.510391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6252.097252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6256.831216] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6262.244272] LustreError: 189520:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6262.266689] LustreError: 189520:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6266.562836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6267.202425] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6271.989839] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6280.296110] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6286.220917] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6293.944533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6294.227729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6297.378106] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6301.617805] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6309.866695] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6322.421523] Lustre: lustre-OST0000: Not available for connect from 192.168.202.33@tcp (stopping) [ 6323.177763] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6323.182984] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6323.188837] LustreError: Skipped 1 previous similar message [ 6323.190601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6323.207357] Lustre: Skipped 1 previous similar message [ 6323.244897] Lustre: Skipped 1 previous similar message [ 6326.029133] Lustre: server umount lustre-OST0000 complete [ 6335.810443] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8d578474f800.ost_server_uuid 50 [ 6337.789946] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6352.434189] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6352.436875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6352.452773] Lustre: Skipped 1 previous similar message [ 6352.471879] Lustre: Skipped 1 previous similar message [ 6357.475040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6357.492365] Lustre: Skipped 1 previous similar message [ 6358.874757] Lustre: server umount lustre-MDT0000 complete [ 6362.593495] LustreError: 189521:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6362.635986] LustreError: 189521:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6362.758204] LustreError: 189501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787398512 with bad export cookie 14252981172216979322 [ 6362.761790] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6362.763299] LustreError: 189501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6363.007481] Lustre: server umount lustre-MDT0001 complete [ 6372.751443] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 07:35:21 (1787398521) [ 6381.150566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6381.744667] LustreError: 193437:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6381.891819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6387.170469] LustreError: 193438:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6388.097400] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6391.271812] LustreError: 193437:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6398.715312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6399.224534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6403.485396] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6412.078828] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6418.674488] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6425.008794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6425.461812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6426.516888] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6432.719396] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6441.513328] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6448.430472] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6448.589086] Lustre: lustre-OST0001: new disk, initializing [ 6448.591329] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6448.594431] Lustre: 196276:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6448.658520] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6454.867553] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6458.956952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6458.965961] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6459.038587] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6464.538703] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6469.607827] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6469.630326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6469.637059] Lustre: Skipped 1 previous similar message [ 6473.798399] Lustre: server umount lustre-OST0000 complete [ 6474.726211] LustreError: 195258:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6474.762077] LustreError: 195258:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6479.892228] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6480.119205] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6484.244872] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6484.473410] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6490.820110] LustreError: 195260:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6490.839918] LustreError: 195260:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 6500.327764] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6500.338420] Lustre: Skipped 1 previous similar message [ 6500.359925] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6500.379507] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6500.381158] Lustre: Skipped 2 previous similar messages [ 6505.204426] Lustre: server umount lustre-OST0001 complete [ 6510.561561] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6510.577200] Lustre: Skipped 1 previous similar message [ 6520.802478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6520.813082] Lustre: Skipped 5 previous similar messages [ 6523.359365] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6523.814382] Lustre: server umount lustre-MDT0000 complete [ 6525.920134] LustreError: 194166:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6525.947846] LustreError: 194166:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 6527.820852] LustreError: 193417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787398677 with bad export cookie 14252981172216980393 [ 6527.834292] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6527.837041] LustreError: 193417:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6528.147692] Lustre: server umount lustre-MDT0001 complete [ 6554.220850] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 07:38:22 (1787398702) [ 6560.946848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6561.147762] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6561.177201] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6561.351380] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6566.576891] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6576.456518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6576.614671] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6576.664327] Lustre: 199726:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6581.822717] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6589.851690] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6595.829262] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6602.917300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6603.100488] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6608.509640] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6609.798644] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6618.129589] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6625.718606] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6625.760345] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6625.927606] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6625.935931] Lustre: Skipped 2 previous similar messages [ 6632.632891] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6634.757928] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6641.093428] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6645.236067] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6645.243157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6645.254864] Lustre: Skipped 2 previous similar messages [ 6645.280428] Lustre: Skipped 1 previous similar message [ 6651.288944] Lustre: server umount lustre-OST0000 complete [ 6652.124709] LustreError: 201885:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6673.504411] Lustre: server umount lustre-OST0001 complete [ 6679.520110] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6679.531357] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6688.369388] Lustre: server umount lustre-MDT0000 complete [ 6691.231288] Lustre: 188831:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787398825/real 1787398825] req@00000000901cd4a0 x1874223053307136/t0(0) o400->lustre-OST0001-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787398841 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6692.319573] Lustre: 188832:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787398825/real 1787398825] req@00000000876fa7fa x1874223053307520/t0(0) o13->lustre-OST0001-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787398841 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6692.484781] LustreError: 199716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787398842 with bad export cookie 14252981172216981800 [ 6692.487147] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6692.495363] LustreError: 199716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6692.746822] Lustre: server umount lustre-MDT0001 complete [ 6717.022708] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 07:41:05 (1787398865) [ 6738.028220] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 6750.431943] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 6792.729848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6792.952235] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6792.974166] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6793.080321] Lustre: lustre-MDT0000: new disk, initializing [ 6793.200161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6793.234787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6798.320184] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6810.484837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6810.645312] Lustre: 207818:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6810.703661] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6810.708300] Lustre: Skipped 1 previous similar message [ 6810.801373] Lustre: lustre-MDT0001: new disk, initializing [ 6810.935700] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6810.949502] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6815.765295] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6827.354844] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6835.583320] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6844.439255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6844.889506] Lustre: lustre-OST0000: new disk, initializing [ 6844.895252] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6844.903705] Lustre: 209790:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6851.156429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6851.168061] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6851.336760] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6852.783814] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6864.733596] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6870.763842] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6871.066482] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6876.485186] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6876.752572] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6882.272658] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6882.275084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6882.288040] Lustre: Skipped 3 previous similar messages [ 6882.293129] Lustre: Skipped 8 previous similar messages [ 6893.025710] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6893.182557] Lustre: server umount lustre-OST0000 complete [ 6898.151816] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6904.143915] Lustre: server umount lustre-MDT0000 complete [ 6907.880802] LustreError: 207831:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6907.912051] LustreError: 207831:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [ 6908.802051] LustreError: 211333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787399058 with bad export cookie 14252981172216983585 [ 6908.813106] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6908.817996] LustreError: 211333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6909.396834] Lustre: server umount lustre-MDT0001 complete [ 6918.836436] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6923.862521] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6932.884456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6937.350058] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6945.920838] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6953.890684] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6962.374848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6970.671398] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 6980.368848] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6986.170491] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6986.489596] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6993.145512] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6993.418943] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6995.939317] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6995.944549] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6995.963655] Lustre: Skipped 2 previous similar messages [ 7001.982399] Lustre: server umount lustre-OST0000 complete [ 7008.894963] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7014.193364] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7044.212337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7049.717101] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7049.745639] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7051.168822] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7059.265938] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7061.400743] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7064.135453] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7064.163324] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7069.190427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7069.204158] Lustre: Skipped 12 previous similar messages [ 7073.168635] Lustre: server umount lustre-OST0000 complete [ 7083.743567] Lustre: server umount lustre-MDT0000 complete [ 7087.305285] LustreError: 211993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787399237 with bad export cookie 14252981172216984467 [ 7087.307398] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7087.314978] LustreError: 211993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7087.883485] Lustre: server umount lustre-MDT0001 complete [ 7096.862862] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 07:47:25 (1787399245) [ 7105.952663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7106.445252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7106.456482] Lustre: Skipped 6 previous similar messages [ 7111.287565] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7120.053654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7124.399604] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7131.759817] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7138.233342] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7145.094344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7153.234685] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7162.574713] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7167.770367] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7167.990781] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7173.647110] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7173.764401] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7181.875597] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7182.100938] Lustre: lustre-OST0001: new disk, initializing [ 7182.105966] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7182.115423] Lustre: 221352:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7188.099022] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7188.109945] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7188.189572] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7188.618560] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7198.019756] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7203.230739] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7203.413816] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7208.103528] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7208.354509] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7214.062367] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7214.065314] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7214.067029] LustreError: Skipped 2 previous similar messages [ 7214.090392] Lustre: Skipped 6 previous similar messages [ 7216.344195] Lustre: server umount lustre-OST0001 complete [ 7219.187347] LustreError: 219802:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7219.207431] LustreError: 219802:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 7222.384852] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7222.614839] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7227.578245] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7227.834579] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7256.317286] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7261.668171] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7261.680892] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7261.689619] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7262.406794] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7270.389211] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7271.901260] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7277.025730] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7288.287338] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7288.423556] Lustre: server umount lustre-OST0001 complete [ 7311.609240] Lustre: server umount lustre-OST0000 complete [ 7321.285403] Lustre: server umount lustre-MDT0000 complete [ 7324.892563] LustreError: 221363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787399474 with bad export cookie 14252981172216985433 [ 7324.898040] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7324.904037] LustreError: 221363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7331.448663] Lustre: server umount lustre-MDT0001 complete [ 7353.189731] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 07:51:41 (1787399501) [ 7359.433444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7359.662222] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7359.713842] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7364.934518] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7373.576464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7373.675708] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7373.714076] Lustre: 226596:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7379.016306] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7386.534554] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7392.082833] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7399.335796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7399.531926] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7406.839393] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7414.178466] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7424.041272] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7424.140425] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7432.392880] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7440.412334] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7443.010481] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7449.924852] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7450.425387] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7456.751932] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7456.971994] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7463.642087] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d5788547800.ost_server_uuid 50 [ 7465.409418] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8d5788547800.ost_server_uuid in IDLE state after 0 sec [ 7466.564565] Lustre: Permanently deactivating lustre-OST0001 [ 7466.579877] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7470.894262] Lustre: Permanently reactivating lustre-OST0001 [ 7470.917958] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7470.923877] Lustre: Skipped 2 previous similar messages [ 7478.767157] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7478.769253] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7478.772613] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7478.772622] Lustre: Skipped 17 previous similar messages [ 7478.792017] Lustre: Skipped 7 previous similar messages [ 7478.822028] LustreError: Skipped 2 previous similar messages [ 7482.888957] Lustre: server umount lustre-OST0001 complete [ 7492.338310] Lustre: server umount lustre-OST0000 complete [ 7502.687932] Lustre: server umount lustre-MDT0000 complete [ 7506.865215] LustreError: 227339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787399656 with bad export cookie 14252981172216986714 [ 7506.866695] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7506.885834] LustreError: 227339:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7507.258374] Lustre: server umount lustre-MDT0001 complete [ 7530.548609] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 07:54:39 (1787399679) [ 7537.855869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7538.066172] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7538.120241] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7538.130830] Lustre: Skipped 2 previous similar messages [ 7542.276371] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7549.650617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7549.720719] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7549.740217] Lustre: 233292:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7554.038755] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7561.385927] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7567.044961] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7574.790457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7574.993566] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7575.026102] Lustre: Permanently deactivating lustre-OST0000 [ 7575.037158] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7580.718061] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7581.256608] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7589.066602] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7597.676030] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7597.742221] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7605.483890] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7606.246306] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7614.570953] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7616.548751] Lustre: Permanently reactivating lustre-OST0000 [ 7616.571605] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7616.582760] Lustre: Skipped 2 previous similar messages [ 7625.277291] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7625.280064] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 45 seconds. I think it's dead, and I am evicting it. exp 00000000106e56a8, cur 1787399775 deadline 1787399760 last 1787399730 [ 7625.298897] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7625.328385] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7625.346000] Lustre: Skipped 1 previous similar message [ 7625.380931] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7636.459271] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7636.473788] LustreError: Skipped 4 previous similar messages [ 7638.514164] Lustre: server umount lustre-OST0001 complete [ 7648.172498] Lustre: server umount lustre-OST0000 complete [ 7658.240331] Lustre: server umount lustre-MDT0000 complete [ 7662.259727] LustreError: 233284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787399812 with bad export cookie 14252981172216988422 [ 7662.270194] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7662.274975] LustreError: 233284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7662.610243] Lustre: server umount lustre-MDT0001 complete [ 7676.137992] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 7678.707506] Key type lgssc unregistered [ 7678.966569] LNet: 238176:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7678.973911] LNetError: 238176:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7680.038239] LNet: Removed LNI 192.168.202.133@tcp [ 7680.920883] Key type .llcrypt unregistered [ 7680.923644] Key type ._llcrypt unregistered [ 7704.823801] Key type ._llcrypt registered [ 7704.826274] Key type .llcrypt registered [ 7704.909080] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 7705.800846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7705.810031] alg: No test for adler32 (adler32-zlib) [ 7707.069446] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 7707.249805] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 7708.919149] Key type lgssc registered [ 7709.976355] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7719.541988] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7719.582418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7721.005348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7724.874760] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7731.171735] LustreError: 239644:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7731.205507] LustreError: 239644:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7732.299964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7732.452386] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7732.477603] Lustre: 239632:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7732.494537] Lustre: Permanently deactivating lustre-MDT0001 [ 7732.504985] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7732.750224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7736.751901] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7739.418988] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7743.956083] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7749.039768] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7755.237898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7755.610716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7757.675403] LustreError: 241486:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7757.688370] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7761.638333] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7762.931421] LustreError: 241488:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7765.993119] LustreError: 241486:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7766.004538] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7766.010031] LustreError: 241486:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7769.318837] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7771.109669] LustreError: 241622:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7771.134342] LustreError: 241622:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7776.627417] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7776.778243] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7781.874602] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7781.888746] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7782.669711] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7789.575898] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7793.613153] Lustre: Permanently reactivating lustre-MDT0001 [ 7793.650143] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7793.658483] Lustre: Skipped 2 previous similar messages [ 7797.474129] Lustre: lustre-MDT0001: Client f266d4ab-93c7-4f42-b2ed-5a4a60594963 (at 192.168.202.33@tcp) reconnecting [ 7800.905138] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7800.914470] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7800.921813] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7800.929852] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7803.962555] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7805.918266] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7812.525103] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7812.793436] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7817.213529] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7817.536325] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7820.389212] Lustre: Permanently deactivating lustre-MDT0001 [ 7820.401849] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7820.405901] Lustre: Skipped 2 previous similar messages [ 7828.127813] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7835.166357] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7835.186929] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7835.213477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7838.178459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7838.185485] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7838.196592] Lustre: Skipped 1 previous similar message [ 7838.215719] Lustre: Skipped 1 previous similar message [ 7840.331838] Lustre: server umount lustre-MDT0000 complete [ 7843.296936] LustreError: 239645:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7843.326426] LustreError: 239645:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 7843.958703] LustreError: 239626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787399993 with bad export cookie 17673375962104095695 [ 7843.965587] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7843.975757] LustreError: 239626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7844.385090] Lustre: server umount lustre-MDT0001 complete [ 7850.256351] Lustre: server umount lustre-OST0000 complete [ 7862.765115] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 08:00:11 (1787400011) [ 7872.273592] Lustre: server umount lustre-OST0001 complete [ 7879.888505] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 7889.279280] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 7931.869401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7932.119991] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7932.130172] Lustre: Skipped 2 previous similar messages [ 7932.174587] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7932.301925] Lustre: lustre-MDT0000: new disk, initializing [ 7932.399714] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7932.429602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7937.222712] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7948.104262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7948.215590] Lustre: 247915:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7948.258490] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7948.262814] Lustre: Skipped 1 previous similar message [ 7948.320309] Lustre: lustre-MDT0001: new disk, initializing [ 7948.394437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7948.434094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7948.443919] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7952.629587] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7962.739681] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7969.466235] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7977.002807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7977.269175] Lustre: lustre-OST0000: new disk, initializing [ 7977.273255] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7977.277388] Lustre: 249886:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7977.331466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7983.851310] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 7986.729761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7986.744290] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7986.845106] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7994.302988] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7996.513805] LustreError: 247921:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8005.701715] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8005.854854] Lustre: lustre-OST0001: new disk, initializing [ 8005.860983] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8005.871082] Lustre: 251001:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8006.608064] LustreError: 247921:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8006.651398] LustreError: 249286:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8012.926870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8012.942622] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8013.117433] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8013.281735] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8016.671374] LustreError: 249286:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8023.296699] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8024.848647] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8028.641067] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8028.651088] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8028.674843] Lustre: Skipped 1 previous similar message [ 8028.685591] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8028.693437] Lustre: Skipped 3 previous similar messages [ 8033.098971] Lustre: server umount lustre-OST0001 complete [ 8033.774966] LustreError: 250349:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8033.809903] LustreError: 250349:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8052.200932] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8052.220210] Lustre: Skipped 1 previous similar message [ 8052.227033] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8057.314586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8057.327411] Lustre: Skipped 2 previous similar messages [ 8064.991681] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8065.396832] Lustre: server umount lustre-OST0000 complete [ 8071.138957] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8071.145448] LustreError: Skipped 1 previous similar message [ 8071.148791] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8071.160879] Lustre: Skipped 1 previous similar message [ 8071.173301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8071.177964] Lustre: Skipped 2 previous similar messages [ 8076.816046] Lustre: server umount lustre-MDT0000 complete [ 8077.793185] LustreError: 247927:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8077.822967] LustreError: 247927:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 8081.209258] LustreError: 247908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787400231 with bad export cookie 17673375962104098502 [ 8081.225100] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8087.785463] Lustre: server umount lustre-MDT0001 complete [ 8100.656663] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 8103.498343] Key type lgssc unregistered [ 8103.782493] LNet: 253036:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8103.813122] LNetError: 253036:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8103.833626] LNet: Removed LNI 192.168.202.133@tcp [ 8104.822790] Key type .llcrypt unregistered [ 8104.825201] Key type ._llcrypt unregistered [ 8150.939921] Key type ._llcrypt registered [ 8150.942830] Key type .llcrypt registered [ 8151.040347] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 8151.791299] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8151.817859] alg: No test for adler32 (adler32-zlib) [ 8153.069906] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 8153.407030] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 8155.151410] Key type lgssc registered [ 8156.188656] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8163.171875] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8163.188310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8164.486691] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8164.499735] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8164.657253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8169.696795] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8178.594128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8178.717973] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8178.764458] Lustre: 255246:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8179.059377] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8183.613481] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8190.497653] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8195.751800] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8202.357686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8202.571609] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8202.764890] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8209.500307] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8210.657919] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8216.953670] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8272.354813] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8272.355450] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8272.356894] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8272.361540] LustreError: Skipped 1 previous similar message [ 8272.371654] Lustre: Skipped 1 previous similar message [ 8276.334926] Lustre: server umount lustre-OST0000 complete [ 8280.610514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8294.183892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8294.547116] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8300.586866] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8308.422315] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8312.938142] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8312.957459] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8312.968886] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8349.152166] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8349.155363] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8349.167567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8349.171786] Lustre: Skipped 1 previous similar message [ 8354.275413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8354.289026] Lustre: Skipped 1 previous similar message [ 8354.653385] Lustre: server umount lustre-OST0000 complete [ 8359.401066] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8359.404402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8359.419473] Lustre: Skipped 2 previous similar messages [ 8359.427851] Lustre: Skipped 1 previous similar message [ 8364.513089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8364.520213] Lustre: Skipped 2 previous similar messages [ 8365.603600] Lustre: server umount lustre-MDT0000 complete [ 8369.605650] LustreError: 255238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787400519 with bad export cookie 2183187955402059082 [ 8369.607211] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8369.613218] LustreError: 255238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8370.010658] Lustre: server umount lustre-MDT0001 complete [ 8382.077288] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 8385.747847] Key type lgssc unregistered [ 8386.061652] LNet: 260669:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8386.086509] LNetError: 260669:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8386.121603] LNet: Removed LNI 192.168.202.133@tcp [ 8387.113625] Key type .llcrypt unregistered [ 8387.116700] Key type ._llcrypt unregistered [ 8412.494595] Key type ._llcrypt registered [ 8412.497468] Key type .llcrypt registered [ 8412.596746] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 8414.224981] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8414.267513] alg: No test for adler32 (adler32-zlib) [ 8415.418517] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 8415.688665] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 8417.375243] Key type lgssc registered [ 8418.561216] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8425.835316] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8425.855323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8427.338570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8432.065733] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8437.743521] LustreError: 262086:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8437.770653] LustreError: 262086:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8442.763436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8442.849884] LustreError: 262087:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8443.089819] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8449.002312] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8457.507859] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8464.236904] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8472.474049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8472.876560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8479.432920] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8486.213900] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8497.471151] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8523.233820] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8523.239713] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8523.249552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8523.256321] Lustre: Skipped 1 previous similar message [ 8523.276944] Lustre: Skipped 1 previous similar message [ 8525.874332] Lustre: server umount lustre-OST0000 complete [ 8530.400781] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8530.411762] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8530.430208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8533.475597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8533.477412] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8535.663966] Lustre: server umount lustre-MDT0000 complete [ 8538.596404] LustreError: 264580:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8538.628602] LustreError: 264580:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8539.406779] LustreError: 262069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787400689 with bad export cookie 1921783378935788764 [ 8539.413139] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8539.425453] LustreError: 262069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8539.702831] Lustre: server umount lustre-MDT0001 complete [ 8553.733232] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 8558.924125] Key type lgssc unregistered [ 8559.304619] LNet: 266420:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8559.318123] LNetError: 266420:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8559.335268] LNet: Removed LNI 192.168.202.133@tcp [ 8560.810615] Key type .llcrypt unregistered [ 8560.813942] Key type ._llcrypt unregistered [ 8580.809507] Key type ._llcrypt registered [ 8580.814523] Key type .llcrypt registered [ 8580.951722] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 8581.862259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8581.874502] alg: No test for adler32 (adler32-zlib) [ 8582.973324] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 8583.214335] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 8584.823251] Key type lgssc registered [ 8585.901250] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8594.928624] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8594.944668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8596.538274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8601.249417] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8606.688266] LustreError: 267647:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8606.722142] LustreError: 267647:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8607.720801] LustreError: 267648:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8609.945829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8610.458248] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8614.329639] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8622.706843] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8629.404516] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8636.853479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8637.235043] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8642.380055] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8643.997208] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8652.917249] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8671.210556] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8671.221724] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8671.265700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8671.278886] Lustre: Skipped 1 previous similar message [ 8676.332965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8676.344266] Lustre: Skipped 1 previous similar message [ 8676.563422] Lustre: server umount lustre-OST0000 complete [ 8682.464490] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8682.484759] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8682.493236] Lustre: Skipped 1 previous similar message [ 8682.497177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8686.561247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8686.562229] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8691.691981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8691.701741] Lustre: Skipped 2 previous similar messages [ 8696.287181] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8696.463215] Lustre: server umount lustre-MDT0000 complete [ 8696.805869] LustreError: 267647:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8696.828871] LustreError: 267647:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8700.882133] LustreError: 267629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787400850 with bad export cookie 5149385389884745866 [ 8700.886926] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8700.902803] LustreError: 267629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8701.143590] Lustre: server umount lustre-MDT0001 complete [ 8714.828750] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 8717.627951] Key type lgssc unregistered [ 8717.909960] LNet: 271328:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8717.921306] LNetError: 271328:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8718.950715] LNet: Removed LNI 192.168.202.133@tcp [ 8719.888811] Key type .llcrypt unregistered [ 8719.890904] Key type ._llcrypt unregistered [ 8744.772035] Key type ._llcrypt registered [ 8744.773966] Key type .llcrypt registered [ 8744.866870] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 8745.887118] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8746.092963] alg: No test for adler32 (adler32-zlib) [ 8747.167096] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 8747.368951] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 8749.088705] Key type lgssc registered [ 8750.185760] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8757.265142] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8757.278785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8758.690138] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8763.586472] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8768.992330] LustreError: 272749:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8769.003385] LustreError: 272749:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8774.115748] LustreError: 272750:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8774.229582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8774.744857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8780.774482] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8790.389676] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8797.956300] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8806.962995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8807.205795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8814.701280] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8823.935407] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8829.809377] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8860.640456] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8860.644439] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8860.669607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8860.704600] Lustre: Skipped 1 previous similar message [ 8860.730189] Lustre: Skipped 1 previous similar message [ 8863.492248] Lustre: server umount lustre-OST0000 complete [ 8870.881719] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8870.893326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8870.911149] Lustre: Skipped 1 previous similar message [ 8874.697898] Lustre: server umount lustre-MDT0000 complete [ 8876.001810] LustreError: 273676:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8876.036137] LustreError: 273676:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8879.140248] LustreError: 272728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787401029 with bad export cookie 5846865504512959706 [ 8879.161378] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8879.565380] Lustre: server umount lustre-MDT0001 complete [ 8892.609870] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 8895.450372] Key type lgssc unregistered [ 8895.842183] LNet: 277055:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8895.854502] LNetError: 277055:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8895.867283] LNet: Removed LNI 192.168.202.133@tcp [ 8896.882679] Key type .llcrypt unregistered [ 8896.885901] Key type ._llcrypt unregistered [ 8916.979790] Key type ._llcrypt registered [ 8916.981813] Key type .llcrypt registered [ 8917.106870] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 8918.277634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8918.332563] alg: No test for adler32 (adler32-zlib) [ 8919.659018] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 8920.007539] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 8921.711235] Key type lgssc registered [ 8923.296224] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8931.809354] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8931.829931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8933.444148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8938.972614] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8943.596468] LustreError: 278292:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8943.625630] LustreError: 278292:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8948.716471] LustreError: 278294:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8949.508904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8949.879386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8954.555414] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8963.914689] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8970.891338] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8979.717353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8980.245406] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8987.832493] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 8996.388815] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9003.830261] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 9019.361047] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9019.401486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9019.413682] Lustre: Skipped 1 previous similar message [ 9021.028994] Lustre: server umount lustre-OST0000 complete [ 9026.528811] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9026.534690] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9026.543033] Lustre: Skipped 1 previous similar message [ 9026.548441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9029.600905] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9029.604536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9031.464305] Lustre: server umount lustre-MDT0000 complete [ 9034.722036] LustreError: 278295:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9034.752896] LustreError: 278295:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9036.024410] LustreError: 278258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787401186 with bad export cookie 8322758121381331548 [ 9036.033656] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9036.040210] LustreError: 278258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9036.635663] Lustre: server umount lustre-MDT0001 complete [ 9049.983415] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 9053.160790] Key type lgssc unregistered [ 9053.500481] LNet: 282020:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9053.519110] LNetError: 282020:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9053.534418] LNet: Removed LNI 192.168.202.133@tcp [ 9054.364374] Key type .llcrypt unregistered [ 9054.366664] Key type ._llcrypt unregistered [ 9069.712973] Key type ._llcrypt registered [ 9069.715884] Key type .llcrypt registered [ 9069.882806] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9090.063804] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 9108.037859] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 9109.404328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9109.477368] alg: No test for adler32 (adler32-zlib) [ 9110.514660] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 9110.828890] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 9112.519300] Key type lgssc registered [ 9113.799878] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9153.157482] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9153.193402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9154.472184] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9154.503461] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9154.575435] Lustre: lustre-MDT0000: new disk, initializing [ 9154.644558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9154.659555] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9159.872832] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 9171.081160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9171.244325] Lustre: 285577:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9171.299668] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9171.303676] Lustre: Skipped 1 previous similar message [ 9171.378646] Lustre: lustre-MDT0001: new disk, initializing [ 9171.488570] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9171.519808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9171.524977] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9177.946453] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 9188.108570] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9194.759690] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9203.898328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9204.306213] Lustre: lustre-OST0000: new disk, initializing [ 9204.309765] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9204.320079] Lustre: 287544:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9204.447957] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9210.994464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9211.045297] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9211.311965] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9212.456910] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 9225.002604] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9229.918190] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9230.122725] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9235.320597] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9235.547693] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9242.087867] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9242.103291] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9242.116569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9243.756539] Lustre: server umount lustre-OST0000 complete [ 9252.323367] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9252.334559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9252.339164] Lustre: Skipped 2 previous similar messages [ 9252.359133] Lustre: Skipped 1 previous similar message [ 9254.720573] Lustre: server umount lustre-MDT0000 complete [ 9258.500782] LustreError: 285567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787401408 with bad export cookie 620390006157727151 [ 9258.517456] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9258.518062] LustreError: 285567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9258.888153] Lustre: server umount lustre-MDT0001 complete [ 9271.343585] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 08:23:39 (1787401419) [ 9281.293911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9281.922750] LustreError: 289961:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9281.950791] LustreError: 289961:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9282.004991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9287.137846] LustreError: 289962:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9289.333787] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 9291.243503] LustreError: 289961:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9296.389251] LustreError: 289962:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9299.254718] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9299.793131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9305.017924] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 9313.947746] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9320.788125] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9328.734593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9329.225693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9337.740202] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 9347.146696] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9364.962719] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9364.980554] LustreError: Skipped 1 previous similar message [ 9364.990158] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9365.034151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9365.046437] Lustre: Skipped 1 previous similar message [ 9367.022680] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9367.056654] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9367.085643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9367.093420] Lustre: Skipped 1 previous similar message [ 9369.872658] Lustre: server umount lustre-OST0000 complete [ 9375.989625] Lustre: server umount lustre-MDT0000 complete [ 9376.741067] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9376.752840] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9376.777672] LustreError: 289962:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9380.300741] LustreError: 289942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787401530 with bad export cookie 620390006157728040 [ 9380.322432] LustreError: 289942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9380.322588] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9380.827912] Lustre: server umount lustre-MDT0001 complete [ 9395.391527] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 9398.065757] Key type lgssc unregistered [ 9398.329921] LNet: 293527:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9398.341923] LNetError: 293527:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9398.361755] LNet: Removed LNI 192.168.202.133@tcp [ 9399.540133] Key type .llcrypt unregistered [ 9399.542308] Key type ._llcrypt unregistered [ 9431.320682] Key type ._llcrypt registered [ 9431.323607] Key type .llcrypt registered [ 9431.438691] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 9448.726224] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 9450.339162] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9450.447638] alg: No test for adler32 (adler32-zlib) [ 9451.641131] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 9452.006416] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 9453.755107] Key type lgssc registered [ 9456.166110] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9514.444094] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9514.479473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9515.756802] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9515.780117] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9515.839333] Lustre: lustre-MDT0000: new disk, initializing [ 9515.899883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9515.915781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9521.099122] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 9532.107157] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9532.249501] Lustre: 297357:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9532.389574] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9532.398150] Lustre: Skipped 1 previous similar message [ 9532.571225] Lustre: lustre-MDT0001: new disk, initializing [ 9532.704060] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9532.752331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9532.779493] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9538.292172] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 9548.382433] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9554.522886] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9561.958828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9562.314973] Lustre: lustre-OST2710: new disk, initializing [ 9562.326821] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9562.337143] Lustre: 299326:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9562.465630] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9568.835051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9568.852209] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9569.021770] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9569.065892] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 9578.964570] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 9586.255681] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9586.511831] Lustre: lustre-OST03e8: new disk, initializing [ 9586.515953] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9586.528070] Lustre: 300387:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9586.617235] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9593.504784] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 9596.522406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9596.532085] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9596.621321] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9602.875160] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 9611.698583] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9612.003638] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9617.579298] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9617.818363] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9622.971331] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9623.170436] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9628.679546] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9628.870938] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9634.969990] Lustre: server umount lustre-MDT0000 complete [ 9635.301408] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9635.320374] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9637.871662] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9637.900965] Lustre: Skipped 2 previous similar messages [ 9639.296241] LustreError: 297347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787401789 with bad export cookie 11653739261870340944 [ 9639.327693] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9639.329572] LustreError: 297347:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9639.813707] Lustre: server umount lustre-MDT0001 complete [ 9654.558891] Lustre: server umount lustre-OST2710 complete [ 9668.871894] Lustre: server umount lustre-OST03e8 complete [ 9689.966851] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [ 9704.243886] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 9748.716259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9748.917574] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9748.940815] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9749.003951] Lustre: lustre-MDT0000: new disk, initializing [ 9749.044393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9749.057374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9752.789473] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 9764.105287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9764.197798] Lustre: 305659:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9764.268559] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9764.272592] Lustre: Skipped 1 previous similar message [ 9764.389523] Lustre: lustre-MDT0001: new disk, initializing [ 9764.513438] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9764.566249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9764.586092] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9769.698939] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 9780.993726] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9787.311858] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9794.669198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9794.959070] Lustre: lustre-OST0000: new disk, initializing [ 9794.966152] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9794.976433] Lustre: 307630:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9795.111642] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9800.762522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9800.778610] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9800.877334] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9802.287551] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 9811.918756] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9817.674651] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9818.020283] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9822.816105] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9823.031923] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9826.790055] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9826.791885] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9826.799681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9826.816779] LustreError: Skipped 1 previous similar message [ 9826.870371] Lustre: Skipped 1 previous similar message [ 9831.433812] Lustre: server umount lustre-OST0000 complete [ 9837.027779] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9837.033294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9837.050151] Lustre: Skipped 1 previous similar message [ 9837.064424] Lustre: Skipped 2 previous similar messages [ 9842.144707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9842.157754] Lustre: Skipped 1 previous similar message [ 9842.915293] Lustre: server umount lustre-MDT0000 complete [ 9846.245952] LustreError: 306462:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9846.268782] LustreError: 306462:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 9847.181739] LustreError: 306456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787401997 with bad export cookie 11653739261870342624 [ 9847.184419] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9847.195458] LustreError: 306456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9847.530097] Lustre: server umount lustre-MDT0001 complete [ 9856.601186] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 08:33:25 (1787402005) [ 9858.189784] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9860.129846] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 08:33:28 (1787402008) [ 9864.061439] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 9893.644557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9893.973820] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9894.028138] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9894.342971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9899.146026] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 9908.828755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9909.053088] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9909.103050] Lustre: 311486:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9914.210768] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [ 9936.895701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9937.191132] Lustre: Denying initial registration attempt from nid 192.168.202.133@tcp, specified as failover [ 9937.209449] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9937.228756] LustreError: 312937:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9937.234745] LustreError: 312937:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9937.246557] LustreError: 312937:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9937.357568] Lustre: server umount lustre-OST0000 complete [ 9937.359776] LustreError: 312937:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9945.061049] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9945.074234] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9945.089560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9951.021084] Lustre: server umount lustre-MDT0000 complete [ 9952.251764] LustreError: 312942:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9952.272294] LustreError: 312942:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9955.603786] LustreError: 311478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787402105 with bad export cookie 11653739261870343506 [ 9955.609731] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9955.627360] LustreError: 311478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9956.244279] Lustre: server umount lustre-MDT0001 complete [ 9968.627439] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [ 9971.993404] Key type lgssc unregistered [ 9972.332860] LNet: 314061:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9972.341954] LNetError: 314061:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9972.357579] LNet: Removed LNI 192.168.202.133@tcp [ 9973.444561] Key type .llcrypt unregistered [ 9973.450602] Key type ._llcrypt unregistered [ 9987.742189] Key type ._llcrypt registered [ 9987.744279] Key type .llcrypt registered [ 9987.849040] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [ 9988.924387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9988.947211] alg: No test for adler32 (adler32-zlib) [ 9989.973432] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [ 9990.253581] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [ 9992.023262] Key type lgssc registered [ 9993.782436] Lustre: Echo OBD driver; http://www.lustre.org/ [10049.832866] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [10058.046527] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10058.058673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10059.422525] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10059.466218] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10059.698103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10064.230111] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [10075.561887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10075.775174] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10075.829437] Lustre: 316827:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10076.324898] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10082.111826] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [10090.325473] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10097.484381] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10118.523192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10118.741430] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10118.947286] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10124.721309] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [10132.418989] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10136.544975] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10136.558055] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10136.597591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10137.057988] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10137.080420] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10137.112591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10137.123260] Lustre: Skipped 1 previous similar message [10140.901249] Lustre: server umount lustre-OST0000 complete [10147.302354] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10147.304481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10147.319791] Lustre: Skipped 1 previous similar message [10147.340494] Lustre: Skipped 1 previous similar message [10151.339677] Lustre: server umount lustre-MDT0000 complete [10155.598923] LustreError: 316821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787402305 with bad export cookie 3036229682853700185 [10155.605149] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10155.609152] LustreError: 316821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10156.099815] Lustre: server umount lustre-MDT0001 complete [10170.462799] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [10173.963222] Key type lgssc unregistered [10174.246961] LNet: 320404:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10174.252393] LNetError: 320404:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10174.290256] LNet: Removed LNI 192.168.202.133@tcp [10175.160517] Key type .llcrypt unregistered [10175.171757] Key type ._llcrypt unregistered [10205.271108] Key type ._llcrypt registered [10205.273816] Key type .llcrypt registered [10205.359868] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [10206.844738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10206.915391] alg: No test for adler32 (adler32-zlib) [10207.989423] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [10208.210974] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [10209.943206] Key type lgssc registered [10210.965718] Lustre: Echo OBD driver; http://www.lustre.org/ [10220.547745] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10220.573906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10222.215897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10228.542793] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [10232.809323] LustreError: 321823:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10232.827310] LustreError: 321823:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10237.929386] LustreError: 321822:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10238.469471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10238.676630] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10243.426070] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [10251.089409] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10259.118302] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10269.809181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10270.342190] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10277.279808] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [10285.940379] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10305.509771] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10305.517761] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10305.540503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10307.835168] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.33@tcp (stopping) [10308.585199] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10308.608848] Lustre: Skipped 1 previous similar message [10311.080499] Lustre: server umount lustre-MDT0000 complete [10312.899159] LustreError: 321819:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10315.100801] LustreError: 321802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787402465 with bad export cookie 4783126304210849156 [10315.102371] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10315.113292] LustreError: 321802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10318.053601] LustreError: 321818:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10318.063847] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.33@tcp (stopping) [10318.070247] LustreError: 321818:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10318.085936] Lustre: Skipped 3 previous similar messages [10318.822505] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10320.868646] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10320.878822] Lustre: Skipped 1 previous similar message [10321.558795] Lustre: server umount lustre-MDT0001 complete [10328.415736] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10338.147146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10341.407844] LustreError: 321132:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f4670fba x1874227214989696/t0(0) o250->MGC192.168.202.133@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10341.756992] LustreError: 323655:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10341.780635] LustreError: 323655:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10341.886294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10341.994043] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10347.220398] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [10351.075922] LustreError: 325436:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10351.087210] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10351.101735] LustreError: 325436:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10358.025410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10358.675412] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10363.888862] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10364.290795] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [10372.515592] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10374.443699] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10381.050560] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10383.312221] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10393.057680] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10393.066320] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10393.094317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10394.606253] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10395.984264] Lustre: server umount lustre-OST0000 complete [10404.832697] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10404.858865] Lustre: Skipped 1 previous similar message [10404.874522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10404.885835] Lustre: Skipped 3 previous similar messages [10408.320103] Lustre: server umount lustre-MDT0000 complete [10409.965739] LustreError: 325443:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10409.986734] LustreError: 325443:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [10412.270458] LustreError: 322557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787402562 with bad export cookie 4783126304210856898 [10412.277202] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10412.287303] LustreError: 322557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10412.774619] Lustre: server umount lustre-MDT0001 complete [10424.293788] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [10427.058678] Key type lgssc unregistered [10427.414627] LNet: 328114:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10427.431624] LNetError: 328114:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10427.467156] LNet: Removed LNI 192.168.202.133@tcp [10428.382518] Key type .llcrypt unregistered [10428.385968] Key type ._llcrypt unregistered [10456.602225] Key type ._llcrypt registered [10456.605145] Key type .llcrypt registered [10456.706858] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [10458.222768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10458.258210] alg: No test for adler32 (adler32-zlib) [10459.327825] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [10459.588139] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [10461.320914] Key type lgssc registered [10462.399928] Lustre: Echo OBD driver; http://www.lustre.org/ [10470.362711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10470.386621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10472.103949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10477.402028] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [10482.680355] LustreError: 329532:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10482.697829] LustreError: 329532:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10487.782643] LustreError: 329531:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10489.611283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10490.650711] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10498.233806] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [10507.491967] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10514.301542] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10521.573090] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10521.579785] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10521.588335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10522.593692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10522.596830] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10524.364634] Lustre: server umount lustre-MDT0000 complete [10527.714779] LustreError: 329531:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10527.734914] LustreError: 329531:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10528.606297] LustreError: 329510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787402678 with bad export cookie 15543395058613806073 [10528.608209] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10528.615535] LustreError: 329510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10528.919834] Lustre: server umount lustre-MDT0001 complete [10544.680780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10545.210623] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10545.266601] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10545.600036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10550.538190] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [10560.508423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10560.666825] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10560.709984] Lustre: 332120:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10561.055423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10565.972274] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [10575.953698] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10587.239272] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10598.973450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10599.302322] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10599.316988] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10599.331473] LustreError: 334005:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [10599.346528] LustreError: 334005:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10599.355864] LustreError: 334005:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10599.442485] Lustre: server umount lustre-OST0000 complete [10599.450863] LustreError: 334005:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [10610.673033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10610.986169] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10611.393921] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10618.556460] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10618.958924] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [10626.732316] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10635.343035] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10635.473753] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10635.549428] Lustre: lustre-OST0001: new disk, initializing [10635.555635] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10635.562913] Lustre: 335549:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10635.683323] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10642.774117] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [10643.047995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10643.057155] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10643.183284] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10651.709679] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [10658.793983] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10658.811615] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10658.842308] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10658.853772] Lustre: Skipped 2 previous similar messages [10661.299098] Lustre: server umount lustre-OST0001 complete [10663.907134] LustreError: 334472:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10663.951990] LustreError: 334472:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10669.033710] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10669.042987] LustreError: 336161:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10669.052375] Lustre: Skipped 1 previous similar message [10669.055959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10669.080030] LustreError: 336161:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10671.966977] Lustre: server umount lustre-OST0000 complete [10678.752593] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10678.755900] LustreError: Skipped 1 previous similar message [10678.772489] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10678.801777] Lustre: Skipped 1 previous similar message [10678.816279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10678.830357] Lustre: Skipped 1 previous similar message [10683.151201] Lustre: server umount lustre-MDT0000 complete [10684.385276] LustreError: 332131:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10684.409951] LustreError: 332131:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10687.386900] LustreError: 332111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787402837 with bad export cookie 15543395058613806626 [10687.403202] LustreError: 332111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10687.406628] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10687.791664] Lustre: server umount lustre-MDT0001 complete [10700.898362] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [10704.245539] Key type lgssc unregistered [10704.547423] LNet: 337510:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10704.555350] LNetError: 337510:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10704.594417] LNet: Removed LNI 192.168.202.133@tcp [10705.634236] Key type .llcrypt unregistered [10705.638770] Key type ._llcrypt unregistered [10790.506348] Key type ._llcrypt registered [10790.510347] Key type .llcrypt registered [10790.672656] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [10809.778512] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [10811.675254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10811.728288] alg: No test for adler32 (adler32-zlib) [10813.021093] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [10813.477426] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [10815.207194] Key type lgssc registered [10816.208657] Lustre: Echo OBD driver; http://www.lustre.org/ [10860.384866] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10860.439664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10861.858140] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10861.913569] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10861.973508] Lustre: lustre-MDT0000: new disk, initializing [10862.072207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10862.088463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10867.477915] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [10880.311393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10880.444311] Lustre: 342453:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10880.508514] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10880.511854] Lustre: Skipped 1 previous similar message [10880.587570] Lustre: lustre-MDT0001: new disk, initializing [10880.689221] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10880.723687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10880.743622] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10885.339972] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [10895.414086] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10901.431407] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10910.300348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10910.764235] Lustre: lustre-OST0000: new disk, initializing [10910.770630] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10910.781519] Lustre: 344421:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10910.898179] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10916.382543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10916.407382] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10916.527172] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10917.397089] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [10928.900419] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10935.697394] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10936.029789] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10942.302556] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10942.582910] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10947.552190] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10947.553048] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10947.564993] LustreError: Skipped 1 previous similar message [10947.567214] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10947.580341] Lustre: Skipped 1 previous similar message [10947.598720] Lustre: Skipped 1 previous similar message [10951.055818] Lustre: server umount lustre-OST0000 complete [10957.801589] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10957.815490] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10957.818471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10957.835638] Lustre: Skipped 1 previous similar message [10961.854348] Lustre: server umount lustre-MDT0000 complete [10965.690542] LustreError: 342445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787403115 with bad export cookie 12793053198046213848 [10965.694337] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10965.705256] LustreError: 342445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10966.414710] Lustre: server umount lustre-MDT0001 complete [10977.507765] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 08:52:06 (1787403126) [10986.348514] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 08:52:14 (1787403134) [10995.357223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10995.702027] LustreError: 347177:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10995.721785] LustreError: 347177:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10995.774405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11000.395309] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [11000.807984] LustreError: 347178:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11005.922351] LustreError: 347177:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11009.301827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11009.540509] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11014.571362] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [11022.781967] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11029.835257] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11039.328060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11039.734588] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11047.110857] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [11056.246082] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11065.260575] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [11067.394175] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [11068.996552] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [11070.614431] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [11072.491228] Lustre: Failing over lustre-MDT0000 [11072.840405] Lustre: server umount lustre-MDT0000 complete [11074.540183] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11074.548035] LustreError: 347172:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11074.564282] Lustre: Skipped 2 previous similar messages [11074.597629] LustreError: 347172:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11079.648127] LustreError: 348162:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11079.695234] LustreError: 348162:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11089.894434] LustreError: 347178:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11089.918409] LustreError: 347178:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [11090.922996] Lustre: 340676:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787403224/real 1787403224] req@00000000c5008fd1 x1874227849403776/t0(0) o400->MGC192.168.202.133@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787403240 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11090.978547] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11096.213182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11101.157676] LustreError: 340674:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000000dd1bff0 x1874227849409536/t0(0) o250->MGC192.168.202.133@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11101.579941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11101.628318] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11105.473255] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11105.889595] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [11106.803667] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11106.840396] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [11106.897570] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11116.367431] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11118.393356] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11123.059809] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [11125.556588] Lustre: Failing over lustre-MDT0000 [11125.866506] Lustre: server umount lustre-MDT0000 complete [11126.061057] LustreError: 347172:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11126.082651] LustreError: 347172:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [11127.265766] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11127.267986] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11127.291312] Lustre: Skipped 2 previous similar messages [11143.648355] Lustre: 340676:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787403277/real 1787403277] req@00000000c0574ef2 x1874227849424896/t0(0) o400->MGC192.168.202.133@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787403293 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11143.669503] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11146.557868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11154.302493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11154.407293] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11157.699039] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11159.528108] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11159.537045] Lustre: Skipped 2 previous similar messages [11159.588314] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [11159.598607] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [11159.669404] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11171.175683] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11173.480603] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11183.025321] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11183.056063] Lustre: Skipped 2 previous similar messages [11183.067538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11183.070199] Lustre: Skipped 3 previous similar messages [11188.194421] LustreError: 347173:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11188.224400] LustreError: 347173:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 29 previous similar messages [11188.255536] Lustre: server umount lustre-MDT0000 complete [11193.546697] LustreError: 351144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787403343 with bad export cookie 12793053198046216564 [11193.550718] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11193.577669] LustreError: 351144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11194.111439] Lustre: server umount lustre-MDT0001 complete [11210.041631] Lustre: server umount lustre-OST0000 complete [11229.040586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11229.751532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11235.039740] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [11245.104672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11251.729118] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [11261.180459] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11267.867319] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11275.913453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11283.094871] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [11291.073161] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11299.978904] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11311.584349] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11311.608373] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11311.642602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11315.695304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11315.716781] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11317.401606] Lustre: server umount lustre-OST0000 complete [11325.923151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11325.937316] Lustre: Skipped 3 previous similar messages [11328.504838] Lustre: server umount lustre-MDT0000 complete [11331.043533] LustreError: 353514:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11331.054626] LustreError: 353514:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [11333.739080] LustreError: 353499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787403483 with bad export cookie 12793053198046217306 [11333.753616] LustreError: 353499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11333.757416] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11334.188044] Lustre: server umount lustre-MDT0001 complete [11346.846316] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [11349.962356] Key type lgssc unregistered [11350.341607] LNet: 357075:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11350.359536] LNetError: 357075:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11350.373890] LNet: Removed LNI 192.168.202.133@tcp [11351.382553] Key type .llcrypt unregistered [11351.385927] Key type ._llcrypt unregistered [11380.811475] Key type ._llcrypt registered [11380.813852] Key type .llcrypt registered [11380.948220] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [11382.213899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11382.251725] alg: No test for adler32 (adler32-zlib) [11383.304614] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [11383.579712] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [11385.335339] Key type lgssc registered [11386.948364] Lustre: Echo OBD driver; http://www.lustre.org/ [11395.040480] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11395.080721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11396.624240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11401.328673] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [11406.820817] LustreError: 358600:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11406.834443] LustreError: 358600:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11410.196678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11410.575067] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11415.591837] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [11424.598298] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11430.062441] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11436.455943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11436.853928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11441.992764] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11443.590473] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [11451.514423] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11466.726486] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11466.733828] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11466.747365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11467.239632] Lustre: server umount lustre-MDT0000 complete [11470.820252] LustreError: 359327:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11470.840794] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11471.433324] LustreError: 358578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787403621 with bad export cookie 5648325950849286867 [11471.436974] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11471.456104] LustreError: 358578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11471.880521] Lustre: server umount lustre-MDT0001 complete [11486.697887] Lustre: server umount lustre-OST0000 complete [11501.655231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11502.057750] LustreError: 362330:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11502.074466] LustreError: 362330:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11502.162193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11506.028517] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [11507.172089] LustreError: 362331:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11512.304195] LustreError: 362330:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11517.175980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11517.864241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11524.483317] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [11533.418313] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11540.577437] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11549.249813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11549.836656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11557.788357] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [11566.379615] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11572.362725] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11588.068796] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11588.078823] Lustre: Skipped 2 previous similar messages [11588.079436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11588.088128] Lustre: Skipped 2 previous similar messages [11591.903101] Lustre: server umount lustre-MDT0000 complete [11593.197981] LustreError: 362331:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11593.221906] LustreError: 362331:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11596.044704] LustreError: 362313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787403746 with bad export cookie 5648325950849287980 [11596.051961] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11596.056425] LustreError: 362313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11596.401302] Lustre: server umount lustre-MDT0001 complete [11611.047219] Lustre: server umount lustre-OST0000 complete [11629.018642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11629.347865] LustreError: 366255:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11629.394108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11633.341423] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [11642.566022] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11648.340692] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [11656.091544] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11661.674367] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11669.958513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11670.369661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11670.378943] Lustre: Skipped 1 previous similar message [11674.513592] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11679.027313] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [11688.660942] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11695.080634] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11695.085470] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11695.094523] Lustre: Skipped 1 previous similar message [11695.101334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11698.150935] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11698.152237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11698.183711] Lustre: Skipped 1 previous similar message [11700.531517] Lustre: server umount lustre-OST0000 complete [11708.385516] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11708.391455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11708.400429] Lustre: Skipped 2 previous similar messages [11708.411315] Lustre: Skipped 1 previous similar message [11718.624785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11718.638350] Lustre: Skipped 2 previous similar messages [11720.159209] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11720.396691] Lustre: server umount lustre-MDT0000 complete [11723.745868] LustreError: 366256:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11723.767552] LustreError: 366256:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11724.538610] LustreError: 366236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787403874 with bad export cookie 5648325950849289002 [11724.546116] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11724.551440] LustreError: 366236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11725.055252] Lustre: server umount lustre-MDT0001 complete [11737.672403] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [11740.157590] Key type lgssc unregistered [11740.382990] LNet: 369790:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11740.387337] LNetError: 369790:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11740.408714] LNet: Removed LNI 192.168.202.133@tcp [11741.296758] Key type .llcrypt unregistered [11741.299535] Key type ._llcrypt unregistered [11766.380805] Key type ._llcrypt registered [11766.382990] Key type .llcrypt registered [11766.492898] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [11767.796297] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11767.810675] alg: No test for adler32 (adler32-zlib) [11768.941654] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [11769.246272] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [11770.944375] Key type lgssc registered [11772.145625] Lustre: Echo OBD driver; http://www.lustre.org/ [11779.314846] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11779.341319] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11779.346195] LustreError: 371237:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11779.359559] LustreError: 371237:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11779.371353] LustreError: 371237:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11779.381728] LustreError: 371237:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11779.387225] LustreError: 371237:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [11787.323418] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11787.329844] LustreError: 371634:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11787.352556] LustreError: 371634:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11787.357210] LustreError: 371634:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11787.361920] LustreError: 371634:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11787.370978] LustreError: 371634:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [11808.019554] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [11810.215257] Key type lgssc unregistered [11810.448729] LNet: 372636:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11810.452227] LNetError: 372636:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11810.472658] LNet: Removed LNI 192.168.202.133@tcp [11811.041434] Key type .llcrypt unregistered [11811.043621] Key type ._llcrypt unregistered [11829.132870] Key type ._llcrypt registered [11829.133977] Key type .llcrypt registered [11829.185585] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [11845.132590] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [11846.032700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11846.094738] alg: No test for adler32 (adler32-zlib) [11847.151168] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [11847.423439] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [11849.071351] Key type lgssc registered [11849.973755] Lustre: Echo OBD driver; http://www.lustre.org/ [11885.904825] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11885.942826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11887.359116] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11887.399845] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11887.521947] Lustre: lustre-MDT0000: new disk, initializing [11887.585223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11887.604637] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11892.063213] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [11907.177630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11907.375742] Lustre: 375808:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11907.469444] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11907.480087] Lustre: Skipped 1 previous similar message [11907.583575] Lustre: lustre-MDT0001: new disk, initializing [11907.797332] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11907.851513] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11907.874911] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11915.093967] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [11926.593403] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11934.093525] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11942.659032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11943.031813] Lustre: lustre-OST0000: new disk, initializing [11943.035631] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11943.044573] Lustre: 377780:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11943.175585] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11949.127526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11949.140599] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11949.336773] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11950.986912] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [11960.704849] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11965.955878] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11966.115637] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11970.409497] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11970.593143] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11975.153804] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11975.169951] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11975.173769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11975.199504] LustreError: Skipped 1 previous similar message [11975.236937] Lustre: Skipped 1 previous similar message [11978.960784] Lustre: server umount lustre-OST0000 complete [11984.353495] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11984.364458] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11984.380277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11984.386153] Lustre: Skipped 1 previous similar message [11985.378572] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11985.397602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11985.404094] Lustre: Skipped 1 previous similar message [11989.520464] Lustre: server umount lustre-MDT0000 complete [11993.065807] LustreError: 375802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787404143 with bad export cookie 4146268367625602232 [11993.072560] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11993.090259] LustreError: 375802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11993.467118] Lustre: server umount lustre-MDT0001 complete [12002.708857] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [12004.946988] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 09:09:13 (1787404153) [12011.890392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12012.230985] LustreError: 380239:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12012.260363] LustreError: 380239:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12012.305231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12017.166582] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [12017.633627] LustreError: 380240:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12022.766272] LustreError: 380239:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12027.381678] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12027.632413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12032.316534] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [12040.304658] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12047.050218] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12055.341102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12055.698661] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12063.033766] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [12071.301516] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12079.936462] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12080.204533] Lustre: lustre-OST0001: new disk, initializing [12080.208256] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12080.219753] Lustre: 383078:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12080.356853] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12085.837024] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12085.859917] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12086.012910] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12086.536478] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [12095.074569] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [12101.610204] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12101.613856] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12101.645054] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12101.651044] Lustre: Skipped 1 previous similar message [12105.011573] Lustre: server umount lustre-OST0001 complete [12106.728215] LustreError: 382062:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12106.753961] LustreError: 382062:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12111.846926] LustreError: 382060:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12111.864653] LustreError: 382060:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12122.086159] LustreError: 382060:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12122.090435] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12122.103849] LustreError: 382060:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12122.150685] Lustre: Skipped 1 previous similar message [12122.157449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12122.162124] Lustre: Skipped 1 previous similar message [12127.561150] Lustre: server umount lustre-OST0000 complete [12132.320809] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12132.321773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12132.342095] Lustre: Skipped 2 previous similar messages [12132.356384] Lustre: Skipped 4 previous similar messages [12137.857604] Lustre: server umount lustre-MDT0000 complete [12141.417655] LustreError: 383076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787404291 with bad export cookie 4146268367625603114 [12141.421477] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12141.429660] LustreError: 383076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12147.940358] Lustre: server umount lustre-MDT0001 complete [12160.346407] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [12162.817654] Key type lgssc unregistered [12163.058424] LNet: 385068:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12163.071993] LNetError: 385068:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12163.103466] LNet: Removed LNI 192.168.202.133@tcp [12163.892468] Key type .llcrypt unregistered [12163.894712] Key type ._llcrypt unregistered [12219.269248] Key type ._llcrypt registered [12219.272038] Key type .llcrypt registered [12219.441887] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [12220.785427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12220.799116] alg: No test for adler32 (adler32-zlib) [12222.044067] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [12222.296445] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [12223.967274] Key type lgssc registered [12225.258616] Lustre: Echo OBD driver; http://www.lustre.org/ [12235.192773] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12235.231357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12236.523577] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12236.573599] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12236.738858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12241.739504] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [12252.040384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12252.216853] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12252.256866] Lustre: 387871:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12252.534560] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12257.649547] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [12265.052864] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12270.688462] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12273.636385] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12273.657651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12278.243228] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12278.259046] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12278.284160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12279.835251] Lustre: server umount lustre-MDT0000 complete [12283.672739] LustreError: 387861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787404433 with bad export cookie 14908971344916485841 [12283.674656] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12283.679568] LustreError: 387861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12284.004973] Lustre: server umount lustre-MDT0001 complete [12293.553664] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 09:14:02 (1787404442) [12300.028747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12300.544256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12304.803739] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [12305.895479] LustreError: 390382:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12305.915943] LustreError: 390382:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12311.014512] LustreError: 390383:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12313.662346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12314.191127] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12319.041464] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [12327.897566] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12333.976835] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12341.107864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12341.310514] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12341.602651] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12348.350534] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [12356.760692] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12365.375110] Lustre: Permanently deactivating lustre-OST0000 [12365.410983] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12373.029887] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12375.658533] LustreError: 393367:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12375.681754] LustreError: 393367:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [12375.689721] LustreError: 393367:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12378.979971] LustreError: 393416:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [12378.986302] LustreError: 393416:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12387.366959] Lustre: server umount lustre-OST0000 complete [12389.920579] LustreError: 393665:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [12389.927634] LustreError: 393665:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12393.446167] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12393.463302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12393.480349] Lustre: Skipped 2 previous similar messages [12398.563497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12398.573534] Lustre: Skipped 1 previous similar message [12401.858619] Lustre: server umount lustre-MDT0000 complete [12403.691237] LustreError: 391108:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12403.723743] LustreError: 391108:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12406.676948] LustreError: 393072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787404556 with bad export cookie 14908971344916486422 [12406.685533] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12406.704529] LustreError: 393072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12417.312091] Lustre: server umount lustre-MDT0001 complete [12427.884709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12432.737656] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [12435.716952] LustreError: 394828:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12443.356317] Lustre: 395167:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.133@tcp [12446.409667] Lustre: server umount MGS complete [12460.065112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12460.528841] LustreError: 395892:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12460.563505] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12460.567036] Lustre: Skipped 1 previous similar message [12460.623796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12465.109300] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [12465.642790] LustreError: 395893:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12475.874755] LustreError: 395892:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12475.902984] LustreError: 395892:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12476.235723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12476.827979] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12476.963262] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12482.121417] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [12490.887586] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12498.103027] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12499.932349] Lustre: Permanently reactivating lustre-OST0000 [12499.969558] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12499.980180] Lustre: Skipped 2 previous similar messages [12529.633147] LustreError: 397720:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12536.428771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12536.770951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12542.765598] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [12550.154696] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12560.361766] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12560.372429] Lustre: Skipped 1 previous similar message [12560.376181] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12560.378661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12560.407514] Lustre: Skipped 1 previous similar message [12562.174695] Lustre: server umount lustre-OST0000 complete [12569.056096] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12569.062343] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12569.076458] Lustre: Skipped 1 previous similar message [12569.086700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12580.319210] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12580.553991] Lustre: server umount lustre-MDT0000 complete [12580.834906] LustreError: 395892:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12580.851496] LustreError: 395892:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12584.688897] LustreError: 395874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787404734 with bad export cookie 14908971344916487472 [12584.698393] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12584.702297] LustreError: 395874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12585.155395] Lustre: server umount lustre-MDT0001 complete [12596.873810] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [12599.300798] Key type lgssc unregistered [12599.578894] LNet: 400303:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12599.589593] LNetError: 400303:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12600.620939] LNet: Removed LNI 192.168.202.133@tcp [12601.371524] Key type .llcrypt unregistered [12601.374066] Key type ._llcrypt unregistered [12617.899191] Key type ._llcrypt registered [12617.901365] Key type .llcrypt registered [12618.002985] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [12633.185941] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [12634.617825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12634.679132] alg: No test for adler32 (adler32-zlib) [12635.779161] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [12636.069681] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [12637.743284] Key type lgssc registered [12638.880667] Lustre: Echo OBD driver; http://www.lustre.org/ [12682.730884] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12682.784814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12684.178694] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12684.212352] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12684.272446] Lustre: lustre-MDT0000: new disk, initializing [12684.371032] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12684.414901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12688.661596] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [12700.433502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12700.580594] Lustre: 403462:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12700.651313] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12700.656829] Lustre: Skipped 1 previous similar message [12700.720397] Lustre: lustre-MDT0001: new disk, initializing [12700.863666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12700.902435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12700.917533] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12705.871182] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [12716.250336] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12722.563763] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12729.799573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12730.022209] Lustre: lustre-OST0000: new disk, initializing [12730.026209] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12730.031555] Lustre: 405428:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12730.111808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12735.971162] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [12739.123665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12739.137093] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12739.239402] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12746.178302] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12750.883472] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12751.056898] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12755.609474] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12755.968109] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12759.579727] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12759.586435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12759.600030] Lustre: Skipped 1 previous similar message [12759.622837] Lustre: Skipped 1 previous similar message [12764.645494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12764.655373] Lustre: Skipped 1 previous similar message [12769.767558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12769.775926] Lustre: Skipped 1 previous similar message [12772.320535] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12772.613459] Lustre: server umount lustre-OST0000 complete [12777.962074] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12777.972617] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12777.993382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12780.003799] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12782.802484] Lustre: server umount lustre-MDT0000 complete [12786.758857] LustreError: 403452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787404936 with bad export cookie 13353910431686865065 [12786.760733] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12786.786555] LustreError: 403452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12787.362081] Lustre: server umount lustre-MDT0001 complete [12797.613891] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12799.564934] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 09:22:28 (1787404948) [12806.951202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12807.509875] LustreError: 407891:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12807.533411] LustreError: 407891:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [12807.593517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12812.368852] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [12812.768109] LustreError: 407892:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12817.890325] LustreError: 407891:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12820.657375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12820.979591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12825.884330] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [12833.580949] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12840.534525] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12848.016853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12848.259990] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12855.445322] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [12863.144705] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12864.746255] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12867.476061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12873.188440] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12873.197522] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12873.209753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12873.219684] Lustre: Skipped 2 previous similar messages [12878.518466] Lustre: server umount lustre-OST0000 complete [12882.400508] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12882.413643] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12882.434914] Lustre: Skipped 1 previous similar message [12882.449758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12882.461886] Lustre: Skipped 2 previous similar messages [12896.223261] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12896.433723] Lustre: server umount lustre-MDT0000 complete [12896.756027] LustreError: 407891:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12896.772891] LustreError: 407891:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12900.586762] LustreError: 407874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787405050 with bad export cookie 13353910431686865954 [12900.588709] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12900.619255] LustreError: 407874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12901.066967] Lustre: server umount lustre-MDT0001 complete [12915.429353] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [12918.990715] Key type lgssc unregistered [12919.260515] LNet: 411619:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12919.270173] LNetError: 411619:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12919.299908] LNet: Removed LNI 192.168.202.133@tcp [12920.106833] Key type .llcrypt unregistered [12920.109910] Key type ._llcrypt unregistered [12948.989428] Key type ._llcrypt registered [12949.006804] Key type .llcrypt registered [12949.226508] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [12949.602357] Key type .llcrypt unregistered [12949.605834] Key type ._llcrypt unregistered [12970.829941] Key type ._llcrypt registered [12970.832544] Key type .llcrypt registered [12970.954994] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [12972.457961] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12972.485089] alg: No test for adler32 (adler32-zlib) [12973.718794] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [12974.027339] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [12975.801456] Key type lgssc registered [12977.448628] Lustre: Echo OBD driver; http://www.lustre.org/ [12986.602625] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12986.632717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12988.178844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12993.818792] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [12998.625073] LustreError: 413839:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12998.643587] LustreError: 413839:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13003.258027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13003.670383] LustreError: 413840:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13003.715243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13008.866490] LustreError: 413839:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13008.878190] LustreError: 413839:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13010.338390] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [13013.991602] LustreError: 413840:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13014.003990] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [13014.021054] LustreError: 413840:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13019.105383] LustreError: 413839:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13019.136068] LustreError: 413839:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13019.295843] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13027.262093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13027.720779] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13032.879147] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [13043.177129] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13043.180822] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13043.194744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13043.209241] Lustre: Skipped 1 previous similar message [13043.216926] Lustre: Skipped 1 previous similar message [13043.381779] Lustre: server umount lustre-OST0000 complete [13048.295178] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13048.305985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13048.314860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13048.332212] Lustre: Skipped 1 previous similar message [13048.358363] Lustre: Skipped 1 previous similar message [13049.951117] Lustre: server umount lustre-MDT0000 complete [13053.415073] LustreError: 413839:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13053.436621] LustreError: 413839:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [13054.035950] LustreError: 413818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787405204 with bad export cookie 774061561975447535 [13054.057169] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13054.085716] LustreError: 413818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13054.559901] Lustre: server umount lustre-MDT0001 complete [13067.783845] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [13070.722979] Key type lgssc unregistered [13071.009957] LNet: 416989:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13071.018720] LNetError: 416989:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13071.046618] LNet: Removed LNI 192.168.202.133@tcp [13071.714312] Key type .llcrypt unregistered [13071.720070] Key type ._llcrypt unregistered [13101.386683] Key type ._llcrypt registered [13101.391283] Key type .llcrypt registered [13101.524798] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [13102.934237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13102.959762] alg: No test for adler32 (adler32-zlib) [13103.957502] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [13104.258684] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [13105.919256] Key type lgssc registered [13106.971801] Lustre: Echo OBD driver; http://www.lustre.org/ [13114.920359] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13114.938054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13130.463744] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13135.584052] LustreError: 418380:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.133@tcp: failed processing log, type 4: rc = -110 [13162.272287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13168.673354] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [13177.777460] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13186.143472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13186.797807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13186.889490] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13192.166959] LustreError: 418405:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13192.173992] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [13192.188749] LustreError: 418405:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13197.291215] LustreError: 418406:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13197.319937] LustreError: 418406:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13202.408844] LustreError: 419416:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13203.140714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13203.679822] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13208.910453] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [13219.870737] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13226.543614] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13238.241077] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13238.251305] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13238.266793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13239.776801] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13239.778541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13239.790057] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13239.830414] Lustre: Skipped 1 previous similar message [13242.785305] Lustre: server umount lustre-OST0000 complete [13248.495925] Lustre: server umount lustre-MDT0000 complete [13250.023312] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13250.033062] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13250.061870] LustreError: 419417:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13250.099993] LustreError: 419417:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13253.889447] LustreError: 418388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787405403 with bad export cookie 1814740857234139494 [13253.896154] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13253.921624] LustreError: 418388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13254.821456] Lustre: server umount lustre-MDT0001 complete [13268.807447] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [13273.225776] Key type lgssc unregistered [13273.601026] LNet: 421955:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13273.607995] LNetError: 421955:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13273.638148] LNet: Removed LNI 192.168.202.133@tcp [13275.106641] Key type .llcrypt unregistered [13275.115434] Key type ._llcrypt unregistered [13301.453842] Key type ._llcrypt registered [13301.456289] Key type .llcrypt registered [13301.536339] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [13302.933202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13303.007786] alg: No test for adler32 (adler32-zlib) [13304.110959] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [13304.395349] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [13306.071616] Key type lgssc registered [13307.664196] Lustre: Echo OBD driver; http://www.lustre.org/ [13316.366652] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13316.383895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13317.950833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13322.057972] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [13327.330364] LustreError: 423376:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13327.356200] LustreError: 423376:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13331.312243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13331.613373] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13336.369896] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [13344.384355] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13351.060240] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13358.236116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13358.698232] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13362.815625] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13365.453258] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [13374.565300] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13381.604503] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13381.608152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13381.623383] Lustre: Skipped 2 previous similar messages [13381.640569] Lustre: Skipped 2 previous similar messages [13382.877832] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.33@tcp (stopping) [13385.906735] Lustre: server umount lustre-MDT0000 complete [13386.721066] LustreError: 423371:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13386.741768] LustreError: 423371:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13387.993224] LustreError: 423370:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13388.504823] LustreError: 423356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.33@tcp arrived at 1787405538 with bad export cookie 1015585627783211850 [13390.467263] LustreError: 423354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787405540 with bad export cookie 1015585627783210954 [13390.480856] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13390.483092] LustreError: 423354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13391.839934] LustreError: 423376:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13391.840144] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13391.871171] LustreError: 423376:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13391.919243] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13391.924285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13395.872936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13395.876836] LustreError: 423375:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13395.893788] LustreError: 423375:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13396.994451] Lustre: server umount lustre-OST0000 complete [13405.644768] Lustre: server umount lustre-MDT0001 complete [13422.357809] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [13426.354107] Key type lgssc unregistered [13426.601027] LNet: 427062:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13426.606252] LNetError: 427062:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13426.638567] LNet: Removed LNI 192.168.202.133@tcp [13427.746476] Key type .llcrypt unregistered [13427.748525] Key type ._llcrypt unregistered [13459.606719] Key type ._llcrypt registered [13459.609363] Key type .llcrypt registered [13459.780301] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [13461.746722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13461.851584] alg: No test for adler32 (adler32-zlib) [13463.023432] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [13463.291500] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [13465.031417] Key type lgssc registered [13466.108739] Lustre: Echo OBD driver; http://www.lustre.org/ [13474.277534] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13474.294677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13475.741817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13480.449298] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [13485.031871] LustreError: 428482:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13485.057407] LustreError: 428482:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13489.653243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13490.061399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13494.894864] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [13502.918972] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13508.852159] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13515.476462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13515.796314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13520.957519] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13523.678515] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [13531.992820] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13539.313814] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13539.332227] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13540.063802] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.33@tcp (stopping) [13540.078512] Lustre: Skipped 1 previous similar message [13543.374445] Lustre: server umount lustre-MDT0001 complete [13544.421973] LustreError: 428481:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13544.441679] LustreError: 428481:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13549.547540] LustreError: 428481:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13549.563372] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13549.571824] LustreError: 428481:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13549.593348] Lustre: Skipped 1 previous similar message [13549.598826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13554.660801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13554.661398] LustreError: 428482:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13555.199312] Lustre: server umount lustre-OST0000 complete [13559.552551] Lustre: server umount lustre-MDT0000 complete [13574.565889] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [13576.906608] Key type lgssc unregistered [13577.214981] LNet: 432164:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13577.223726] LNetError: 432164:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13578.279455] LNet: Removed LNI 192.168.202.133@tcp [13579.252318] Key type .llcrypt unregistered [13579.256324] Key type ._llcrypt unregistered [13608.179415] Key type ._llcrypt registered [13608.181809] Key type .llcrypt registered [13608.328944] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [13623.267409] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [13624.530422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13624.619622] alg: No test for adler32 (adler32-zlib) [13625.859604] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [13626.177593] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [13627.839247] Key type lgssc registered [13629.024833] Lustre: Echo OBD driver; http://www.lustre.org/ [13671.510825] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [13689.196561] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13689.215751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13690.547366] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13690.607186] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13690.769875] Lustre: lustre-MDT0000: new disk, initializing [13690.866601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13690.883175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13696.600067] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [13715.686416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13716.144145] Lustre: lustre-OST0000: new disk, initializing [13716.147852] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13716.150871] Lustre: Skipped 1 previous similar message [13716.157793] Lustre: 437199:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13716.248391] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13722.197590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13722.212148] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13722.237095] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13722.793247] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [13734.956217] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13749.874255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13750.084785] Lustre: 436183:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13750.152974] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13750.237657] Lustre: lustre-MDT0001: new disk, initializing [13750.352368] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13750.422948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13750.442427] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13756.810760] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [13769.728927] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13770.092601] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13774.629449] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13774.772154] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13782.500119] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13782.507970] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13782.527332] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13782.534653] Lustre: Skipped 1 previous similar message [13783.272719] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.33@tcp (stopping) [13787.623502] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13787.642474] Lustre: Skipped 1 previous similar message [13792.753569] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13792.767048] Lustre: Skipped 2 previous similar messages [13793.248471] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13793.484260] Lustre: server umount lustre-MDT0001 complete [13802.981389] LustreError: 436194:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13802.995519] LustreError: 436194:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [13808.098113] LustreError: 436189:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13808.122122] LustreError: 436189:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13813.217337] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13813.217609] LustreError: 436194:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13813.236269] Lustre: Skipped 1 previous similar message [13813.239984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13818.333513] Lustre: server umount lustre-OST0000 complete [13818.366215] LustreError: 436193:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13837.281473] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13837.767132] Lustre: server umount lustre-MDT0000 complete [13856.446822] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [13860.380585] Key type lgssc unregistered [13860.677239] LNet: 440951:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13860.700856] LNetError: 440951:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13860.717964] LNet: Removed LNI 192.168.202.133@tcp [13862.012760] Key type .llcrypt unregistered [13862.014319] Key type ._llcrypt unregistered [13913.432765] Key type ._llcrypt registered [13913.434565] Key type .llcrypt registered [13913.526285] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [13913.874380] Key type .llcrypt unregistered [13913.876558] Key type ._llcrypt unregistered [13935.614705] Key type ._llcrypt registered [13935.617271] Key type .llcrypt registered [13935.742626] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [13936.984401] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13937.046954] alg: No test for adler32 (adler32-zlib) [13938.043710] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [13938.227100] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [13939.839181] Key type lgssc registered [13941.059721] Lustre: Echo OBD driver; http://www.lustre.org/ [13972.645416] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13972.709370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13974.081668] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13974.112120] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13974.227562] Lustre: lustre-MDT0000: new disk, initializing [13974.353390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13974.378984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13979.157591] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [13992.329897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13992.465325] Lustre: 444330:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13992.569869] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13992.578586] Lustre: Skipped 1 previous similar message [13992.709485] Lustre: lustre-MDT0001: new disk, initializing [13992.835479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13992.904544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13992.917478] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13998.070792] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [14009.757715] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14016.958568] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14025.272179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14025.621474] Lustre: lustre-OST0000: new disk, initializing [14025.626914] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14025.634265] Lustre: 446300:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14025.726216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14033.256855] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [14033.524383] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14033.548252] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14033.695254] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14044.509388] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14054.373801] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14054.394225] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14054.397758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14054.425945] Lustre: Skipped 2 previous similar messages [14056.892353] Lustre: server umount lustre-MDT0000 complete [14060.897127] LustreError: 446310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787406210 with bad export cookie 11590580038858561933 [14060.909467] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14061.526746] Lustre: server umount lustre-MDT0001 complete [14075.512897] Lustre: server umount lustre-OST0000 complete [14088.496970] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 09:43:56 (1787406236) [14098.502825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14099.083068] LustreError: 448331:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14099.102333] LustreError: 448331:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [14099.158533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14104.466167] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [14104.555123] LustreError: 448332:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14109.671479] LustreError: 448331:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14114.784121] LustreError: 448332:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14115.258572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14115.655208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14120.947302] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [14129.584185] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14136.477507] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14157.736192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14158.079914] Lustre: Found index 0 for lustre-OST0000, updating log [14158.405480] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14165.188362] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [14170.716647] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14174.888487] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14180.832433] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14180.843362] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14180.846747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14180.888208] Lustre: Skipped 3 previous similar messages [14181.940574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14181.947107] Lustre: Skipped 1 previous similar message [14186.138826] Lustre: server umount lustre-OST0000 complete [14203.360858] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14203.371588] Lustre: Skipped 1 previous similar message [14203.371712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14203.383603] Lustre: Skipped 2 previous similar messages [14208.480996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14208.484600] Lustre: Skipped 1 previous similar message [14209.512390] LustreError: 448328:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14209.529268] LustreError: 448328:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [14209.562921] Lustre: server umount lustre-MDT0000 complete [14213.102241] LustreError: 448312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787406363 with bad export cookie 11590580038858563333 [14213.108250] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14213.109742] LustreError: 448312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [14213.275418] Lustre: server umount lustre-MDT0001 complete [14223.567346] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 09:46:11 (1787406371) [14227.999543] LNet: 1 local NIs in recovery (showing 1): 192.168.202.133@tcp [14854.111238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14854.502192] LustreError: 454744:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14854.562330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14858.956932] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [14859.752634] LustreError: 454745:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14864.881150] LustreError: 454744:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14867.991083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14868.436644] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14873.371673] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [14880.820374] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14886.535576] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14906.976072] LNet: 1 local NIs in recovery (showing 1): 192.168.202.133@tcp [14907.574765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14907.882469] Lustre: Found index 0 for lustre-OST0000, updating log [14908.229542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14914.282811] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [14921.757456] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14926.822519] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14926.845134] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14926.846262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14930.312782] Lustre: server umount lustre-OST0000 complete [14936.419538] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14936.424856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14936.436056] Lustre: Skipped 1 previous similar message [14938.535278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14938.549727] Lustre: Skipped 4 previous similar messages [14941.677105] LustreError: 455473:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14941.687623] LustreError: 455473:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [14941.757950] Lustre: server umount lustre-MDT0000 complete [14945.453332] LustreError: 456597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787407095 with bad export cookie 11590580038858564418 [14945.460578] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14945.820056] Lustre: server umount lustre-MDT0001 complete [14967.392146] LNet: 1 local NIs in recovery (showing 1): 192.168.202.133@tcp [14978.685865] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [14989.579368] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [15026.495227] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 09:59:35 (1787407175) [15045.304926] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [15047.809261] Key type lgssc unregistered [15048.211214] LNet: 461359:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15048.215842] LNetError: 461359:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15169.061920] LNet: Removed LNI 192.168.202.133@tcp [15170.079133] Key type .llcrypt unregistered [15170.082436] Key type ._llcrypt unregistered [15191.424561] Key type ._llcrypt registered [15191.430208] Key type .llcrypt registered [15191.525995] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [15193.000516] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15193.054349] alg: No test for adler32 (adler32-zlib) [15194.130661] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [15194.404303] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [15196.119207] Key type lgssc registered [15197.514543] Lustre: Echo OBD driver; http://www.lustre.org/ [15202.231554] LNet: Added LNI 192.168.202.133@tcp100 [8/256/0/180] [15203.239419] LNet: Added LNI 192.168.202.133@tcp101 [8/256/0/180] [15204.076475] LNet: Added LNI 192.168.202.133@tcp102 [8/256/0/180] [15205.095382] LNet: Added LNI 192.168.202.133@tcp103 [8/256/0/180] [15205.924384] LNet: Added LNI 192.168.202.133@tcp104 [8/256/0/180] [15207.239593] LNet: Added LNI 192.168.202.133@tcp105 [8/256/0/180] [15208.328386] LNet: Added LNI 192.168.202.133@tcp106 [8/256/0/180] [15209.455504] LNet: Added LNI 192.168.202.133@tcp107 [8/256/0/180] [15210.538798] LNet: Added LNI 192.168.202.133@tcp108 [8/256/0/180] [15211.885038] LNet: Added LNI 192.168.202.133@tcp109 [8/256/0/180] [15213.171904] LNet: Added LNI 192.168.202.133@tcp110 [8/256/0/180] [15214.696332] LNet: Added LNI 192.168.202.133@tcp111 [8/256/0/180] [15215.868502] LNet: Added LNI 192.168.202.133@tcp112 [8/256/0/180] [15217.014021] LNet: Added LNI 192.168.202.133@tcp113 [8/256/0/180] [15218.051804] LNet: Added LNI 192.168.202.133@tcp114 [8/256/0/180] [15219.321079] LNet: Added LNI 192.168.202.133@tcp115 [8/256/0/180] [15220.304970] LNet: Added LNI 192.168.202.133@tcp116 [8/256/0/180] [15221.354276] LNet: Added LNI 192.168.202.133@tcp117 [8/256/0/180] [15222.316551] LNet: Added LNI 192.168.202.133@tcp118 [8/256/0/180] [15223.495375] LNet: Added LNI 192.168.202.133@tcp119 [8/256/0/180] [15224.520540] LNet: Added LNI 192.168.202.133@tcp120 [8/256/0/180] [15225.489929] LNet: Added LNI 192.168.202.133@tcp121 [8/256/0/180] [15226.371140] LNet: Added LNI 192.168.202.133@tcp122 [8/256/0/180] [15227.291949] LNet: Added LNI 192.168.202.133@tcp123 [8/256/0/180] [15228.386805] LNet: Added LNI 192.168.202.133@tcp124 [8/256/0/180] [15229.366427] LNet: Added LNI 192.168.202.133@tcp125 [8/256/0/180] [15230.275649] LNet: Added LNI 192.168.202.133@tcp126 [8/256/0/180] [15231.091792] LNet: Added LNI 192.168.202.133@tcp127 [8/256/0/180] [15232.131977] LNet: Added LNI 192.168.202.133@tcp128 [8/256/0/180] [15233.157016] LNet: Added LNI 192.168.202.133@tcp129 [8/256/0/180] [15234.030836] LNet: Added LNI 192.168.202.133@tcp130 [8/256/0/180] [15234.879615] LNet: Added LNI 192.168.202.133@tcp131 [8/256/0/180] [15235.958922] LNet: Added LNI 192.168.202.133@tcp132 [8/256/0/180] [15236.967476] LNet: Added LNI 192.168.202.133@tcp133 [8/256/0/180] [15237.824280] LNet: Added LNI 192.168.202.133@tcp134 [8/256/0/180] [15238.758385] LNet: Added LNI 192.168.202.133@tcp135 [8/256/0/180] [15246.723749] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15246.757676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15247.930103] Lustre: 464406:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.133@tcp: can't add NID 192.168.202.133@tcp130: rc = -75 [15248.040635] LNetError: 461861:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.133 [15248.058595] LNetError: Protocol error connecting to 192.168.202.133@tcp117 at host 192.168.202.133:988: is it running a compatible version of Lustre? [15248.080962] LNetError: Skipped 8 previous similar messages [15248.174783] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15248.204330] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15248.285390] Lustre: lustre-MDT0000: new disk, initializing [15248.379317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15248.415462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15248.673899] LNet: 6 local NIs in recovery (showing 6): 192.168.202.133@tcp125, 192.168.202.133@tcp126, 192.168.202.133@tcp127, 192.168.202.133@tcp129, 192.168.202.133@tcp128, 192.168.202.133@tcp117 [15249.119936] LNetError: 464428:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.133 [15249.128292] LNetError: 464428:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 8 previous similar messages [15249.139765] LNetError: Protocol error connecting to 192.168.202.133@tcp111 at host 192.168.202.133:988: is it running a compatible version of Lustre? [15251.757431] LNetError: 461861:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.133 [15251.777863] LNetError: 461861:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 6 previous similar messages [15251.798880] LNetError: Protocol error connecting to 192.168.202.133@tcp125 at host 192.168.202.133:988: is it running a compatible version of Lustre? [15251.818071] LNetError: Skipped 9 previous similar messages [15253.318122] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [15255.842578] LNetError: 461863:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.133 [15255.851877] LNetError: 461863:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 3 previous similar messages [15255.861223] LNetError: Protocol error connecting to 192.168.202.133@tcp125 at host 192.168.202.133:988: is it running a compatible version of Lustre? [15263.008362] LNetError: 461862:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.133 [15263.024526] LNetError: Protocol error connecting to 192.168.202.133@tcp125 at host 192.168.202.133:988: is it running a compatible version of Lustre? [15263.581526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15263.759475] Lustre: 464422:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15263.874379] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15263.887055] Lustre: Skipped 1 previous similar message [15264.036329] Lustre: lustre-MDT0001: new disk, initializing [15264.108211] Lustre: 465223:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.133@tcp: can't add NID 192.168.202.133@tcp130: rc = -75 [15264.125519] Lustre: 465223:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15264.276837] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15264.312295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15264.327499] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15268.880514] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [15271.455892] Lustre: 465587:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.133@tcp: can't add NID 192.168.202.133@tcp130: rc = -75 [15271.469990] Lustre: 465587:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15279.221592] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15279.396156] LNetError: 461864:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.133 [15279.417904] LNetError: Protocol error connecting to 192.168.202.133@tcp125 at host 192.168.202.133:988: is it running a compatible version of Lustre? [15285.134608] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15292.044465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15292.253815] Lustre: lustre-OST0000: new disk, initializing [15292.256868] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15292.262532] Lustre: 466396:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15292.345322] Lustre: 466401:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.133@tcp: can't add NID 192.168.202.133@tcp130: rc = -75 [15292.351186] Lustre: 466401:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15292.391557] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15295.782344] LNetError: 461864:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.133 [15295.795872] LNetError: 461864:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 8 previous similar messages [15295.803746] LNetError: Protocol error connecting to 192.168.202.133@tcp124 at host 192.168.202.133:988: is it running a compatible version of Lustre? [15295.819240] LNetError: Skipped 8 previous similar messages [15298.744780] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [15301.147965] Lustre: 466854:0:(lustre_peer.c:124:class_add_uuid()) 192.168.202.133@tcp: can't add NID 192.168.202.133@tcp130: rc = -75 [15301.156420] Lustre: 466854:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15301.221904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15301.235579] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15301.329473] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15308.065817] LNet: 1 local NIs in recovery (showing 1): 192.168.202.133@tcp125 [15308.644100] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15322.093374] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15322.130555] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15322.155519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15322.182040] Lustre: Skipped 1 previous similar message [15327.201679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15327.220234] Lustre: Skipped 1 previous similar message [15329.574132] LNetError: 461864:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.133 [15329.599605] LNetError: 461864:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 15 previous similar messages [15329.606669] LNetError: Protocol error connecting to 192.168.202.133@tcp123 at host 192.168.202.133:988: is it running a compatible version of Lustre? [15329.636075] LNetError: Skipped 15 previous similar messages [15332.319270] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15332.331791] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15332.361421] Lustre: Skipped 1 previous similar message [15332.699540] Lustre: server umount lustre-OST0000 complete [15337.441231] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15337.458588] Lustre: Skipped 1 previous similar message [15337.465615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15342.562614] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15342.565723] Lustre: Skipped 1 previous similar message [15343.795257] Lustre: server umount lustre-MDT0000 complete [15348.350498] LustreError: 464414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787407498 with bad export cookie 6766695623013798125 [15348.354897] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15348.363343] LustreError: 464414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15348.680576] Lustre: server umount lustre-MDT0001 complete [15362.965848] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [15366.527508] Key type lgssc unregistered [15367.014966] LNet: 468313:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15367.024145] LNetError: 468313:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15367.029945] LNet: Removed LNI 192.168.202.133@tcp [15368.096280] LNet: Removed LNI 192.168.202.133@tcp100 [15368.145797] LNetError: 468313:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15368.160925] LNetError: 468313:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15368.165732] LNet: Removed LNI 192.168.202.133@tcp101 [15369.183288] LNet: Removed LNI 192.168.202.133@tcp102 [15369.186194] LNetError: 468313:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15369.193018] LNetError: 468313:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15369.197597] LNet: Removed LNI 192.168.202.133@tcp103 [15370.208859] LNet: Removed LNI 192.168.202.133@tcp104 [15371.233303] LNet: Removed LNI 192.168.202.133@tcp105 [15371.240713] LNetError: 468313:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15371.249245] LNetError: 468313:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15371.257198] LNet: Removed LNI 192.168.202.133@tcp106 [15372.319182] LNet: Removed LNI 192.168.202.133@tcp107 [15373.343932] LNet: Removed LNI 192.168.202.133@tcp108 [15374.367304] LNet: Removed LNI 192.168.202.133@tcp109 [15375.391768] LNet: Removed LNI 192.168.202.133@tcp110 [15375.403137] LNetError: 468313:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15375.417537] LNetError: 468313:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15375.424084] LNet: Removed LNI 192.168.202.133@tcp111 [15376.479408] LNet: Removed LNI 192.168.202.133@tcp112 [15377.503615] LNet: Removed LNI 192.168.202.133@tcp113 [15378.529843] LNet: Removed LNI 192.168.202.133@tcp114 [15379.551439] LNet: Removed LNI 192.168.202.133@tcp115 [15380.639943] LNet: Removed LNI 192.168.202.133@tcp116 [15381.663212] LNet: Removed LNI 192.168.202.133@tcp117 [15382.688593] LNet: Removed LNI 192.168.202.133@tcp118 [15383.711399] LNet: Removed LNI 192.168.202.133@tcp119 [15383.717981] LNetError: 468313:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15383.739014] LNetError: 468313:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15383.758175] LNet: Removed LNI 192.168.202.133@tcp120 [15384.799363] LNet: Removed LNI 192.168.202.133@tcp121 [15385.824192] LNet: Removed LNI 192.168.202.133@tcp122 [15386.848284] LNet: Removed LNI 192.168.202.133@tcp123 [15387.871858] LNet: Removed LNI 192.168.202.133@tcp124 [15387.874084] LNet: Removed LNI 192.168.202.133@tcp125 [15388.895236] LNet: Removed LNI 192.168.202.133@tcp126 [15389.919321] LNet: Removed LNI 192.168.202.133@tcp127 [15390.943178] LNet: Removed LNI 192.168.202.133@tcp128 [15391.967334] LNet: Removed LNI 192.168.202.133@tcp129 [15391.972411] LNet: Removed LNI 192.168.202.133@tcp130 [15391.978196] LNet: Removed LNI 192.168.202.133@tcp131 [15391.983206] LNet: Removed LNI 192.168.202.133@tcp132 [15391.990987] LNet: Removed LNI 192.168.202.133@tcp133 [15391.993034] LNet: Removed LNI 192.168.202.133@tcp134 [15392.020569] LNet: Removed LNI 192.168.202.133@tcp135 [15393.094092] Key type .llcrypt unregistered [15393.095673] Key type ._llcrypt unregistered [15419.802748] Key type ._llcrypt registered [15419.804456] Key type .llcrypt registered [15419.888192] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [15437.200755] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [15438.807172] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15438.848644] alg: No test for adler32 (adler32-zlib) [15440.069657] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [15440.511815] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [15442.223186] Key type lgssc registered [15443.622077] Lustre: Echo OBD driver; http://www.lustre.org/ [15485.334364] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 10:07:13 (1787407633) [15504.174910] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [15506.914930] Key type lgssc unregistered [15507.290678] LNet: 472403:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15507.302319] LNetError: 472403:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15507.336662] LNet: Removed LNI 192.168.202.133@tcp [15508.131771] Key type .llcrypt unregistered [15508.149417] Key type ._llcrypt unregistered [15527.108752] Key type ._llcrypt registered [15527.111319] Key type .llcrypt registered [15527.212595] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [15528.242579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15528.322368] alg: No test for adler32 (adler32-zlib) [15529.373329] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [15529.587287] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [15531.239716] Key type lgssc registered [15532.275045] Lustre: Echo OBD driver; http://www.lustre.org/ [15541.984254] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15542.027902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15543.282272] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15543.306281] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15543.371620] Lustre: lustre-MDT0000: new disk, initializing [15543.438247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15543.463900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15547.960874] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [15560.087576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15560.275640] Lustre: 473615:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15560.353386] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15560.357623] Lustre: Skipped 1 previous similar message [15560.449639] Lustre: lustre-MDT0001: new disk, initializing [15560.630391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15560.761075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15560.773697] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15565.792959] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [15576.114546] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15581.238984] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15588.781225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15589.047738] Lustre: lustre-OST0000: new disk, initializing [15589.053425] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15589.061828] Lustre: 475589:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15589.158851] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15596.751268] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [15598.182250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15598.191365] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15598.319349] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15606.969618] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15610.884400] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing lnet_if_list [15617.531866] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15617.576746] LNet: Added LNI 192.168.202.133@tcp10 [8/256/0/180] [15624.496342] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15624.535603] LNet: Added LNI 192.168.202.133@tcp11 [8/256/0/180] [15629.243140] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15629.292850] LNet: Added LNI 192.168.202.133@tcp12 [8/256/0/180] [15634.348254] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15634.389950] LNet: Added LNI 192.168.202.133@tcp13 [8/256/0/180] [15639.754902] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15639.796527] LNet: Added LNI 192.168.202.133@tcp14 [8/256/0/180] [15645.052898] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15645.097361] LNet: Added LNI 192.168.202.133@tcp15 [8/256/0/180] [15651.246579] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15651.278183] LNet: Added LNI 192.168.202.133@tcp16 [8/256/0/180] [15656.887431] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15656.919327] LNet: Added LNI 192.168.202.133@tcp17 [8/256/0/180] [15662.170676] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15662.200478] LNet: Added LNI 192.168.202.133@tcp18 [8/256/0/180] [15668.458813] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15668.504934] LNet: Added LNI 192.168.202.133@tcp19 [8/256/0/180] [15674.859287] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15674.913533] LNet: Added LNI 192.168.202.133@tcp20 [8/256/0/180] [15681.292134] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15681.318679] LNet: Added LNI 192.168.202.133@tcp21 [8/256/0/180] [15687.311776] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15687.339393] LNet: Added LNI 192.168.202.133@tcp22 [8/256/0/180] [15693.570983] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15693.603595] LNet: Added LNI 192.168.202.133@tcp23 [8/256/0/180] [15699.061968] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15699.084749] LNet: Added LNI 192.168.202.133@tcp24 [8/256/0/180] [15704.852427] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15704.892668] LNet: Added LNI 192.168.202.133@tcp25 [8/256/0/180] [15711.237136] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15711.278866] LNet: Added LNI 192.168.202.133@tcp26 [8/256/0/180] [15717.145676] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15717.174343] LNet: Added LNI 192.168.202.133@tcp27 [8/256/0/180] [15723.880104] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15723.926025] LNet: Added LNI 192.168.202.133@tcp28 [8/256/0/180] [15730.958279] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15730.987213] LNet: Added LNI 192.168.202.133@tcp29 [8/256/0/180] [15739.439297] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15739.477987] LNet: Added LNI 192.168.202.133@tcp30 [8/256/0/180] [15746.483245] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15746.534277] LNet: Added LNI 192.168.202.133@tcp31 [8/256/0/180] [15754.118189] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15754.154807] LNet: Added LNI 192.168.202.133@tcp32 [8/256/0/180] [15760.656736] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15760.677537] LNet: Added LNI 192.168.202.133@tcp33 [8/256/0/180] [15767.462437] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15767.486350] LNet: Added LNI 192.168.202.133@tcp34 [8/256/0/180] [15774.604851] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15774.632951] LNet: Added LNI 192.168.202.133@tcp35 [8/256/0/180] [15783.095221] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15783.128959] LNet: Added LNI 192.168.202.133@tcp36 [8/256/0/180] [15791.273530] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15791.306238] LNet: Added LNI 192.168.202.133@tcp37 [8/256/0/180] [15799.296683] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15799.330927] LNet: Added LNI 192.168.202.133@tcp38 [8/256/0/180] [15806.244045] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15806.268694] LNet: Added LNI 192.168.202.133@tcp39 [8/256/0/180] [15815.171320] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15815.202050] LNet: Added LNI 192.168.202.133@tcp40 [8/256/0/180] [15824.231105] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15824.261230] LNet: Added LNI 192.168.202.133@tcp41 [8/256/0/180] [15831.842767] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15831.865052] LNet: Added LNI 192.168.202.133@tcp42 [8/256/0/180] [15841.197814] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15841.255241] LNet: Added LNI 192.168.202.133@tcp43 [8/256/0/180] [15849.955612] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15849.984418] LNet: Added LNI 192.168.202.133@tcp44 [8/256/0/180] [15859.367732] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15859.392316] LNet: Added LNI 192.168.202.133@tcp45 [8/256/0/180] [15867.416541] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15867.448286] LNet: Added LNI 192.168.202.133@tcp46 [8/256/0/180] [15875.160583] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15875.195637] LNet: Added LNI 192.168.202.133@tcp47 [8/256/0/180] [15885.493305] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15885.533205] LNet: Added LNI 192.168.202.133@tcp48 [8/256/0/180] [15896.022919] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15896.065293] LNet: Added LNI 192.168.202.133@tcp49 [8/256/0/180] [15906.375794] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15906.419919] LNet: Added LNI 192.168.202.133@tcp50 [8/256/0/180] [15929.312585] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15929.324085] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15929.350340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15931.361918] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15931.363629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15935.518870] Lustre: server umount lustre-MDT0000 complete [15939.740044] LustreError: 475588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787408089 with bad export cookie 11475586721918950244 [15939.741059] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15939.753775] LustreError: 475588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15940.139480] Lustre: server umount lustre-MDT0001 complete [15957.728139] Lustre: 472890:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787408091/real 1787408091] req@0000000075cdde2c x1874232794596864/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787408107 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15957.753580] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15957.765940] Lustre: Skipped 1 previous similar message [15960.036867] Lustre: 472887:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787408094/real 1787408094] req@000000003eff15f8 x1874232794596992/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787408110 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15960.849399] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [15962.081955] Lustre: 472888:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787408096/real 1787408096] req@00000000947af534 x1874232794597120/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787408112 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15965.152446] Lustre: 472889:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787408099/real 1787408099] req@00000000c6748b33 x1874232794597376/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787408115 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15969.090472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15970.335256] Lustre: 472889:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787408104/real 1787408104] req@00000000443258cf x1874232794597504/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787408120 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15984.776758] LustreError: 475581:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15984.794319] LustreError: 475581:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [15984.844808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15986.917616] LustreError: 475582:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15986.933813] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15987.939708] LustreError: 475581:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15989.329404] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [15993.316350] LustreError: 475582:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15997.579117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15997.935537] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16002.027287] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16003.023041] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [16010.748177] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [16016.752478] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [16036.322810] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16036.341085] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16036.378642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16036.385638] Lustre: Skipped 2 previous similar messages [16041.176912] Lustre: server umount lustre-OST0000 complete [16048.099512] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16048.116922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16048.119666] Lustre: Skipped 2 previous similar messages [16048.132654] Lustre: Skipped 3 previous similar messages [16053.220322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16053.229093] Lustre: Skipped 1 previous similar message [16058.337757] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16058.634193] Lustre: server umount lustre-MDT0000 complete [16062.866939] LustreError: 475588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787408212 with bad export cookie 11475586721918952883 [16062.886126] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16062.887632] LustreError: 475588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [16063.617771] Lustre: server umount lustre-MDT0001 complete [16074.668850] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [16077.689428] Key type lgssc unregistered [16078.000471] LNet: 502637:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16078.008512] LNetError: 502637:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [16078.018833] LNet: Removed LNI 192.168.202.133@tcp [16078.022992] LNet: Removed LNI 192.168.202.133@tcp10 [16078.025592] LNet: Removed LNI 192.168.202.133@tcp11 [16078.032934] LNet: Removed LNI 192.168.202.133@tcp12 [16078.036073] LNet: Removed LNI 192.168.202.133@tcp13 [16078.039626] LNet: Removed LNI 192.168.202.133@tcp14 [16078.043421] LNet: Removed LNI 192.168.202.133@tcp15 [16078.045158] LNet: Removed LNI 192.168.202.133@tcp16 [16078.048359] LNet: Removed LNI 192.168.202.133@tcp17 [16078.050812] LNet: Removed LNI 192.168.202.133@tcp18 [16078.052945] LNet: Removed LNI 192.168.202.133@tcp19 [16078.064980] LNet: Removed LNI 192.168.202.133@tcp20 [16078.067426] LNet: Removed LNI 192.168.202.133@tcp21 [16078.069362] LNet: Removed LNI 192.168.202.133@tcp22 [16078.071452] LNet: Removed LNI 192.168.202.133@tcp23 [16078.073621] LNet: Removed LNI 192.168.202.133@tcp24 [16078.075557] LNet: Removed LNI 192.168.202.133@tcp25 [16078.077525] LNet: Removed LNI 192.168.202.133@tcp26 [16078.079492] LNet: Removed LNI 192.168.202.133@tcp27 [16078.081481] LNet: Removed LNI 192.168.202.133@tcp28 [16078.083507] LNet: Removed LNI 192.168.202.133@tcp29 [16078.085601] LNet: Removed LNI 192.168.202.133@tcp30 [16078.087838] LNet: Removed LNI 192.168.202.133@tcp31 [16078.089546] LNet: Removed LNI 192.168.202.133@tcp32 [16078.090881] LNet: Removed LNI 192.168.202.133@tcp33 [16078.092182] LNet: Removed LNI 192.168.202.133@tcp34 [16078.093255] LNet: Removed LNI 192.168.202.133@tcp35 [16078.094650] LNet: Removed LNI 192.168.202.133@tcp36 [16078.095972] LNet: Removed LNI 192.168.202.133@tcp37 [16078.097781] LNet: Removed LNI 192.168.202.133@tcp38 [16078.099454] LNet: Removed LNI 192.168.202.133@tcp39 [16078.101067] LNet: Removed LNI 192.168.202.133@tcp40 [16078.103471] LNet: Removed LNI 192.168.202.133@tcp41 [16078.105287] LNet: Removed LNI 192.168.202.133@tcp42 [16078.107486] LNet: Removed LNI 192.168.202.133@tcp43 [16078.109499] LNet: Removed LNI 192.168.202.133@tcp44 [16078.112117] LNet: Removed LNI 192.168.202.133@tcp45 [16078.114073] LNet: Removed LNI 192.168.202.133@tcp46 [16078.115719] LNet: Removed LNI 192.168.202.133@tcp47 [16078.118333] LNet: Removed LNI 192.168.202.133@tcp48 [16078.120326] LNet: Removed LNI 192.168.202.133@tcp49 [16078.131805] LNet: Removed LNI 192.168.202.133@tcp50 [16078.840156] Key type .llcrypt unregistered [16078.844346] Key type ._llcrypt unregistered [16100.970270] Key type ._llcrypt registered [16100.977837] Key type .llcrypt registered [16101.100361] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [16117.567905] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [16118.788000] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16119.099159] alg: No test for adler32 (adler32-zlib) [16120.286581] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [16120.601669] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [16122.231279] Key type lgssc registered [16123.545815] Lustre: Echo OBD driver; http://www.lustre.org/ [16163.804671] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16163.817991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16165.089850] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16165.117643] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16165.176834] Lustre: lustre-MDT0000: new disk, initializing [16165.230435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16165.248441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16170.506587] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [16182.126898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16182.286806] Lustre: 505837:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16182.331528] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16182.336226] Lustre: Skipped 1 previous similar message [16182.405606] Lustre: lustre-MDT0001: new disk, initializing [16182.540045] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16182.579256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16182.595819] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16186.919761] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [16197.741655] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [16204.927319] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [16212.383223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16212.689857] Lustre: *** cfs_fail_loc=90f, val=0*** [16212.780453] Lustre: lustre-OST0000: new disk, initializing [16212.784720] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16212.790274] Lustre: 507858:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16212.889880] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16220.098375] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [16220.302114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16220.312599] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16220.626677] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16231.537144] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [16246.245958] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16246.255873] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16246.265253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16246.265964] Lustre: Skipped 1 previous similar message [16246.276828] Lustre: Skipped 1 previous similar message [16247.638822] Lustre: server umount lustre-OST0000 complete [16254.432540] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16254.442919] LustreError: Skipped 1 previous similar message [16254.449594] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16254.476549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16256.482304] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16256.490068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16256.518889] Lustre: Skipped 1 previous similar message [16258.357833] Lustre: server umount lustre-MDT0000 complete [16262.749200] LustreError: 505832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787408412 with bad export cookie 3217848704712068125 [16262.757178] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16262.761711] LustreError: 505832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16263.157499] Lustre: server umount lustre-MDT0001 complete [16276.253686] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [16278.952906] Key type lgssc unregistered [16279.217032] LNet: 509764:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16279.231049] LNetError: 509764:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [16279.247371] LNet: Removed LNI 192.168.202.133@tcp [16280.123883] Key type .llcrypt unregistered [16280.129977] Key type ._llcrypt unregistered [16302.196125] Key type ._llcrypt registered [16302.198457] Key type .llcrypt registered [16302.320415] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [16303.401363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16303.485072] alg: No test for adler32 (adler32-zlib) [16304.521845] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [16304.717436] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [16306.383183] Key type lgssc registered [16307.751876] Lustre: Echo OBD driver; http://www.lustre.org/ [16316.007211] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16316.024339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16317.421156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16322.295977] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [16327.651670] LustreError: 511032:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16327.671269] LustreError: 511032:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [16332.071627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16332.513514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16337.883657] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [16347.553983] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [16355.815420] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [16363.674608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16364.187905] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16371.672423] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [16380.684256] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [16397.569384] loop0: detected capacity change from 0 to 204800000 [16397.738833] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16397.751939] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16397.761865] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16404.346279] loop0: detected capacity change from 0 to 204800000 [16404.561041] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16404.576954] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16404.584944] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16409.612426] loop0: detected capacity change from 0 to 204800000 [16414.873831] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16415.045449] Lustre: fs15246-OST0000: new disk, initializing [16415.053508] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16415.061140] Lustre: 514643:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16415.136281] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16422.086329] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [16433.496367] loop1: detected capacity change from 0 to 204800000 [16439.915611] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16440.128110] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16440.193931] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16440.298347] Lustre: fs15246-MDT0000: new disk, initializing [16440.509544] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16440.538205] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16440.697620] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16441.767492] Lustre: 515765:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000008d8f19b1 x1874233607111424/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787408607 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16445.746739] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [16450.855603] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16450.870863] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16450.924501] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16473.579987] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16473.588638] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16477.060345] Lustre: server umount fs15246-MDT0000 complete [16478.705435] LustreError: 512554:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16483.819699] LustreError: 511028:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16486.680209] Lustre: server umount fs15246-OST0000 complete [16500.708295] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16500.710939] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16500.724120] LustreError: Skipped 1 previous similar message [16500.726676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16500.742959] Lustre: Skipped 1 previous similar message [16500.771561] Lustre: Skipped 1 previous similar message [16504.299366] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16504.308550] Lustre: Skipped 1 previous similar message [16506.104924] Lustre: server umount lustre-OST0000 complete [16511.974582] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16511.984248] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16512.013786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16514.530026] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16516.627583] Lustre: server umount lustre-MDT0000 complete [16519.652727] LustreError: 511029:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16519.678348] LustreError: 511029:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [16520.530228] LustreError: 511012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787408670 with bad export cookie 12486560869768044225 [16520.537983] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16520.546315] LustreError: 511012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16521.052307] Lustre: server umount lustre-MDT0001 complete [16532.571728] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [16536.244600] Key type lgssc unregistered [16536.850700] LNet: 518864:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16536.872559] LNetError: 518864:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [16536.922384] LNet: Removed LNI 192.168.202.133@tcp [16537.996972] Key type .llcrypt unregistered [16537.999082] Key type ._llcrypt unregistered [16561.510276] Key type ._llcrypt registered [16561.514655] Key type .llcrypt registered [16561.585380] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [16563.034665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16563.180554] alg: No test for adler32 (adler32-zlib) [16564.244273] Lustre: Lustre: Build Version: 2.17.54_109_gc67c2f0 [16564.485115] LNet: Added LNI 192.168.202.133@tcp [8/256/0/180] [16566.167193] Key type lgssc registered [16567.364686] Lustre: Echo OBD driver; http://www.lustre.org/ [16611.395344] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [16624.099249] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [16668.822060] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16668.886846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16670.188857] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16670.245546] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16670.356946] Lustre: lustre-MDT0000: new disk, initializing [16670.439609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16670.451586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16674.701200] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [16686.259379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16686.423819] Lustre: 523333:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16686.497825] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16686.502543] Lustre: Skipped 1 previous similar message [16686.639169] Lustre: lustre-MDT0001: new disk, initializing [16686.745436] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16686.790405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16686.807188] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16691.772648] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [16702.353757] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [16709.085520] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [16716.185292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16716.622308] Lustre: lustre-OST0000: new disk, initializing [16716.635898] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16716.651782] Lustre: 525305:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16716.859534] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16723.008202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16723.026799] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16723.192941] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16723.674926] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_default_debug -1 all [16734.521397] Lustre: DEBUG MARKER: oleg233-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [16739.761361] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16739.994843] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16744.891340] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16745.089426] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16749.024930] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16749.044733] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16749.049588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16749.061115] Lustre: Skipped 1 previous similar message [16749.099923] Lustre: Skipped 1 previous similar message [16753.727328] Lustre: server umount lustre-OST0000 complete [16758.762088] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16758.776835] LustreError: Skipped 1 previous similar message [16758.779796] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16758.800823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16764.389676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16764.404566] Lustre: Skipped 3 previous similar messages [16769.507407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16773.089018] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16773.350897] Lustre: server umount lustre-MDT0000 complete [16778.458407] LustreError: 523325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787408928 with bad export cookie 10562455667263364297 [16778.463668] LustreError: MGC192.168.202.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16778.473490] LustreError: 523325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [16779.196177] Lustre: server umount lustre-MDT0001 complete [16809.910302] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing set_hostid [16824.970656] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing load_modules_local [16860.451354] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16575 sec ========= 10:30:08 (1787409008) [16862.904613] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 10:30:10 (1787409010) === [16866.920584] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 10:30:15 (1787409015) === [16911.097439] Lustre: DEBUG MARKER: oleg233-server.virtnet: executing unload_modules_local [16914.153571] Key type lgssc unregistered [16914.461845] LNet: 531973:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16914.472320] LNetError: 531973:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [16914.491666] LNet: Removed LNI 192.168.202.133@tcp [16915.433185] Key type .llcrypt unregistered [16915.436993] Key type ._llcrypt unregistered