[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 484471348 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003244] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.008832] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010011] pid_max: default: 32768 minimum: 301 [ 0.011124] LSM: Security Framework initializing [ 0.012050] Yama: becoming mindful. [ 0.013034] SELinux: Initializing. [ 0.014059] *** VALIDATE selinux *** [ 0.022653] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026836] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029097] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030123] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032051] *** VALIDATE tmpfs *** [ 0.033496] *** VALIDATE proc *** [ 0.034254] *** VALIDATE cgroup *** [ 0.035010] *** VALIDATE cgroup2 *** [ 0.036241] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038047] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040032] Spectre V2 : User space: Vulnerable [ 0.041010] Speculative Store Bypass: Vulnerable [ 0.044304] debug: unmapping init [mem 0xffffffffa2c59000-0xffffffffa2c60fff] [ 0.046815] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047737] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048027] ... version: 2 [ 0.049014] ... bit width: 48 [ 0.050014] ... generic registers: 4 [ 0.051014] ... value mask: 0000ffffffffffff [ 0.052013] ... max period: 00007fffffffffff [ 0.053014] ... fixed-purpose events: 3 [ 0.054014] ... event mask: 000000070000000f [ 0.055321] rcu: Hierarchical SRCU implementation. [ 0.057464] smp: Bringing up secondary CPUs ... [ 0.058548] x86: Booting SMP configuration: [ 0.059024] .... node #0, CPUs: #1 #2 #3 [ 0.062171] smp: Brought up 1 node, 4 CPUs [ 0.064013] smpboot: Max logical packages: 1 [ 0.065020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.126112] node 0 deferred pages initialised in 59ms [ 0.129094] devtmpfs: initialized [ 0.130163] x86/mm: Memory block size: 128MB [ 0.132318] gcov: version magic: 0x41383552 [ 0.134117] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.137098] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.139258] pinctrl core: initialized pinctrl subsystem [ 0.140134] [ 0.140517] ************************************************************* [ 0.142009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.145010] ** ** [ 0.147010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.149011] ** ** [ 0.150012] ** This means that this kernel is built to expose internal ** [ 0.153010] ** IOMMU data structures, which may compromise security on ** [ 0.155012] ** your system. ** [ 0.157010] ** ** [ 0.159012] ** If you see this message and you are not debugging the ** [ 0.161010] ** kernel, report this immediately to your vendor! ** [ 0.163008] ** ** [ 0.165007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.167010] ************************************************************* [ 0.169384] NET: Registered protocol family 16 [ 0.172467] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.175220] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.178087] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.181098] cpuidle: using governor menu [ 0.182715] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.183375] PCI: Using configuration type 1 for base access [ 0.184142] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.193074] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.195037] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.198080] cryptd: max_cpu_qlen set to 1000 [ 0.201225] ACPI: Added _OSI(Module Device) [ 0.202000] ACPI: Added _OSI(Processor Device) [ 0.204027] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.206021] ACPI: Added _OSI(Processor Aggregator Device) [ 0.211716] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.219508] ACPI: Interpreter enabled [ 0.221097] ACPI: PM: (supports S0 S3 S4 S5) [ 0.223019] ACPI: Using IOAPIC for interrupt routing [ 0.224134] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.228667] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.239411] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.241040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.244021] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.248081] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.254540] acpiphp: Slot [2] registered [ 0.256189] acpiphp: Slot [5] registered [ 0.257166] acpiphp: Slot [6] registered [ 0.259145] acpiphp: Slot [7] registered [ 0.260146] acpiphp: Slot [8] registered [ 0.262183] acpiphp: Slot [9] registered [ 0.263125] acpiphp: Slot [10] registered [ 0.265145] acpiphp: Slot [3] registered [ 0.266084] acpiphp: Slot [4] registered [ 0.268111] acpiphp: Slot [11] registered [ 0.269098] acpiphp: Slot [12] registered [ 0.270119] acpiphp: Slot [13] registered [ 0.272098] acpiphp: Slot [14] registered [ 0.273144] acpiphp: Slot [15] registered [ 0.275143] acpiphp: Slot [16] registered [ 0.277153] acpiphp: Slot [17] registered [ 0.278128] acpiphp: Slot [18] registered [ 0.279142] acpiphp: Slot [19] registered [ 0.281130] acpiphp: Slot [20] registered [ 0.282173] acpiphp: Slot [21] registered [ 0.283126] acpiphp: Slot [22] registered [ 0.285145] acpiphp: Slot [23] registered [ 0.287134] acpiphp: Slot [24] registered [ 0.288118] acpiphp: Slot [25] registered [ 0.290110] acpiphp: Slot [26] registered [ 0.291000] acpiphp: Slot [27] registered [ 0.291000] acpiphp: Slot [28] registered [ 0.293094] acpiphp: Slot [29] registered [ 0.294118] acpiphp: Slot [30] registered [ 0.296104] acpiphp: Slot [31] registered [ 0.297055] PCI host bridge to bus 0000:00 [ 0.298018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.301021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.303020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.305021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.307032] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.310043] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.312218] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.315159] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.318651] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.329926] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.334612] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.338040] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.340024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.343031] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.346509] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.349860] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.352038] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.356124] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.362013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.373018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.377867] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.382967] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.399029] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.408025] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.429017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.439524] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.447014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.455019] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.466021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.473162] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.478014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.481015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.493014] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.499830] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.505011] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.509016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.522015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.529000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.534012] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.539012] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.550015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.560585] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.565015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.571019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.584013] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.591000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.593252] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.595203] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.596247] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.598136] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.603049] iommu: Default domain type: Passthrough [ 0.605429] SCSI subsystem initialized [ 0.606136] ACPI: bus type USB registered [ 0.607082] usbcore: registered new interface driver usbfs [ 0.609066] usbcore: registered new interface driver hub [ 0.610075] usbcore: registered new device driver usb [ 0.611105] pps_core: LinuxPPS API ver. 1 registered [ 0.613010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.615071] PTP clock support registered [ 0.617125] EDAC MC: Ver: 3.0.0 [ 0.619156] PCI: Using ACPI for IRQ routing [ 0.620861] NetLabel: Initializing [ 0.622009] NetLabel: domain hash size = 128 [ 0.623006] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.625072] NetLabel: unlabeled traffic allowed by default [ 0.627064] vgaarb: loaded [ 0.628161] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.629010] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.636346] clocksource: Switched to clocksource kvm-clock [ 0.740967] VFS: Disk quotas dquot_6.6.0 [ 0.742349] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.744408] *** VALIDATE ramfs *** [ 0.745530] *** VALIDATE hugetlbfs *** [ 0.746832] pnp: PnP ACPI init [ 0.748960] pnp: PnP ACPI: found 6 devices [ 0.766647] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.770050] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.772315] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.774340] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.776834] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.779338] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.782264] NET: Registered protocol family 2 [ 0.784580] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.789120] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.792552] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.797254] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.799953] TCP: Hash tables configured (established 65536 bind 65536) [ 0.802044] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.804407] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.806968] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.808960] NET: Registered protocol family 1 [ 0.811346] RPC: Registered named UNIX socket transport module. [ 0.813543] RPC: Registered udp transport module. [ 0.814688] RPC: Registered tcp transport module. [ 0.815816] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.817360] NET: Registered protocol family 44 [ 0.818410] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.819792] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.821698] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.824105] PCI: CLS 0 bytes, default 64 [ 0.825850] Unpacking initramfs... [ 2.247212] debug: unmapping init [mem 0xffff9a1cbcc54000-0xffff9a1cbffbffff] [ 2.250616] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.252094] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.253990] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.754369] Initialise system trusted keyrings [ 2.755966] Key type blacklist registered [ 2.757865] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.766442] zbud: loaded [ 2.769344] *** VALIDATE nfs *** [ 2.770473] *** VALIDATE nfs4 *** [ 2.771773] pstore: using deflate compression [ 2.774799] Platform Keyring initialized [ 2.863300] NET: Registered protocol family 38 [ 2.865558] Key type asymmetric registered [ 2.867018] Asymmetric key parser 'x509' registered [ 2.868452] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.870974] io scheduler mq-deadline registered [ 2.872372] io scheduler kyber registered [ 2.874022] io scheduler bfq registered [ 2.876216] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.879359] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.882231] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.885336] ACPI: Power Button [PWRF] [ 2.891204] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.897787] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.914075] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.920518] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.934413] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.961446] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.991580] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.998130] Non-volatile memory driver v1.3 [ 2.999855] Linux agpgart interface v0.103 [ 3.032077] virtio_blk virtio1: [vda] 149760 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.035336] vda: detected capacity change from 0 to 76677120 [ 3.047678] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.050576] vdb: detected capacity change from 0 to 1073741824 [ 3.062173] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.064052] vdc: detected capacity change from 0 to 2621440000 [ 3.073709] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.075555] vdd: detected capacity change from 0 to 2621440000 [ 3.087177] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.089051] vde: detected capacity change from 0 to 4294967296 [ 3.103261] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.106510] vdf: detected capacity change from 0 to 4294967296 [ 3.112817] libphy: Fixed MDIO Bus: probed [ 3.129361] usbcore: registered new interface driver usbserial_generic [ 3.131673] usbserial: USB Serial support registered for generic [ 3.133928] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.137753] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.139418] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.141582] mousedev: PS/2 mouse device common for all mice [ 3.144438] rtc_cmos 00:05: RTC can wake from S4 [ 3.147302] rtc_cmos 00:05: registered as rtc0 [ 3.147352] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.148826] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.154315] intel_pstate: CPU model not supported [ 3.157759] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.162402] hid: raw HID events driver (C) Jiri Kosina [ 3.163686] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.164547] usbcore: registered new interface driver usbhid [ 3.168744] usbhid: USB HID core driver [ 3.170461] drop_monitor: Initializing network drop monitor service [ 3.172476] Initializing XFRM netlink socket [ 3.173867] NET: Registered protocol family 10 [ 3.178235] Segment Routing with IPv6 [ 3.179300] NET: Registered protocol family 17 [ 3.180865] mpls_gso: MPLS GSO support [ 3.187145] RAS: Correctable Errors collector initialized. [ 3.189384] AVX version of gcm_enc/dec engaged. [ 3.191190] AES CTR mode by8 optimization enabled [ 3.262805] sched_clock: Marking stable (3262783909, 0)->(4154302430, -891518521) [ 3.266133] registered taskstats version 1 [ 3.268175] Loading compiled-in X.509 certificates [ 3.270193] zswap: loaded using pool lzo/zbud [ 3.293839] Key type big_key registered [ 3.306852] Key type encrypted registered [ 3.308571] ima: No TPM chip found, activating TPM-bypass! [ 3.310847] ima: Allocated hash algorithm: sha1 [ 3.312652] ima: No architecture policies found [ 3.314423] evm: Initialising EVM extended attributes: [ 3.316570] evm: security.selinux [ 3.317882] evm: security.ima [ 3.318985] evm: security.capability [ 3.320421] evm: HMAC attrs: 0x1 [ 3.322787] rtc_cmos 00:05: setting system clock to 2026-09-03 12:50:43 UTC (1788439843) [ 3.328773] debug: unmapping init [mem 0xffffffffa3c03000-0xffffffffa3dfffff] [ 3.331885] debug: unmapping init [mem 0xffffffffa2982000-0xffffffffa2c58fff] [ 3.346089] Write protecting the kernel read-only data: 28672k [ 3.351269] debug: unmapping init [mem 0xffffffffa1003000-0xffffffffa11fffff] [ 3.354339] debug: unmapping init [mem 0xffffffffa1914000-0xffffffffa19fffff] [ 3.384183] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.393380] systemd[1]: Detected virtualization kvm. [ 3.395301] systemd[1]: Detected architecture x86-64. [ 3.397172] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.420098] systemd[1]: No hostname configured. [ 3.422066] systemd[1]: Set hostname to . [ 3.424263] random: systemd: uninitialized urandom read (16 bytes read) [ 3.426760] systemd[1]: Initializing machine ID from random generator. [ 3.469603] random: ln: uninitialized urandom read (6 bytes read) [ 3.551429] random: systemd: uninitialized urandom read (16 bytes read) [ 3.554196] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.558301] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.562761] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.094591] device-mapper: uevent: version 1.0.3 [ 4.096597] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.774116] virtio_net virtio0 ens2: renamed from eth0 [ 4.778131] random: fast init done [ 4.826402] scsi host0: ata_piix [ 4.926744] scsi host1: ata_piix [ 4.928388] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.930563] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.000394] dracut-initqueue[587]: RTNETLINK answers: File exists [ 9.824836] random: crng init done [ 9.826419] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.068917] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.151572] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.406277] SELinux: Disabled at runtime. [ 11.468847] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.478214] systemd[1]: Detected virtualization kvm. [ 11.480147] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.005473] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.009419] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.014473] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.022278] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.026941] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.033456] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.037909] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Stopped target Switch Root. [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Kernel Debug File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Remount Root and Kernel File Systems... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ 12.158461] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.439478] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.774222] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.811646] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.924664] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.934612] EDAC sbridge: Ver: 1.1.2 [ 14.153133] Key type dns_resolver registered [ 14.433597] NFS: Registering the id_resolver key type [ 14.435727] Key type id_resolver registered [ 14.437369] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg127-server login: [ 65.840736] libcfs: loading out-of-tree module taints kernel. [ 65.906079] Key type ._llcrypt registered [ 65.908864] Key type .llcrypt registered [ 66.060696] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 85.634167] hrtimer: interrupt took 4277092 ns [ 90.358854] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 91.974504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 91.984280] alg: No test for adler32 (adler32-zlib) [ 93.368492] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 94.578814] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 96.423218] Key type lgssc registered [ 98.059773] Lustre: Echo OBD driver; http://www.lustre.org/ [ 115.647587] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 162.944836] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 177.240142] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 177.264101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 178.686399] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 178.701343] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 178.783425] Lustre: lustre-MDT0000: new disk, initializing [ 178.857848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 178.875547] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 184.173548] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 200.265723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 200.325691] Lustre: 6515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 200.356150] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 200.358779] Lustre: Skipped 1 previous similar message [ 200.430234] Lustre: lustre-MDT0001: new disk, initializing [ 200.508274] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 200.559733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 200.579658] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 207.810902] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 212.817356] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 223.258283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 223.643625] Lustre: lustre-OST0000: new disk, initializing [ 223.653372] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 223.663914] Lustre: 8455:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 223.785438] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 229.371427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 229.375813] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 229.459847] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 230.516316] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 245.105890] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 245.321346] Lustre: lustre-OST0001: new disk, initializing [ 245.323618] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 245.329889] Lustre: 9528:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 245.423200] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 251.463942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 251.471391] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 251.530894] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 252.996683] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 265.956246] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 276.608399] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 282.631694] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing check_logdir /tmp/testlogs/ [ 289.737893] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing yml_node [ 294.861785] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 297.716885] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 300.958462] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 302.717155] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 08:55:42 EDT 2026 [ 321.699972] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 323.548836] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 344.033604] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 344.035324] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 344.036306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 344.055215] Lustre: Skipped 1 previous similar message [ 345.785984] Lustre: server umount lustre-MDT0000 complete [ 350.900939] LustreError: 7932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440191 with bad export cookie 18255456302599898253 [ 350.904122] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 350.914200] LustreError: 7932:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 354.273676] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 354.280909] LustreError: 6807:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 354.281123] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 354.281126] Lustre: Skipped 3 previous similar messages [ 354.287657] Lustre: Skipped 2 previous similar messages [ 354.315630] LustreError: 6807:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 355.295912] LustreError: 6524:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 355.297360] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 355.310403] LustreError: 6524:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 355.317461] Lustre: Skipped 2 previous similar messages [ 357.613479] Lustre: server umount lustre-MDT0001 complete [ 362.849440] Lustre: server umount lustre-OST0000 complete [ 366.817677] Lustre: server umount lustre-OST0001 complete [ 374.013880] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 384.211701] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 424.532490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 424.825689] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 424.847547] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 424.961569] Lustre: lustre-MDT0000: new disk, initializing [ 425.082887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 425.102335] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 430.833156] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 441.862060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 442.086396] Lustre: 15082:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 442.213409] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 442.225461] Lustre: Skipped 1 previous similar message [ 442.343499] Lustre: lustre-MDT0001: new disk, initializing [ 442.584942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 442.692392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 442.712041] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 448.350341] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 458.506270] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 465.582521] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 474.348174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 474.617955] Lustre: lustre-OST0000: new disk, initializing [ 474.623122] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 474.631676] Lustre: 17050:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 474.718087] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 476.016288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 476.026777] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 476.136371] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 482.813083] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 493.589712] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 499.008158] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 499.197415] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 503.801881] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 503.991248] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 506.851647] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 506.869128] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 506.874631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 506.881827] Lustre: Skipped 1 previous similar message [ 511.924560] Lustre: server umount lustre-OST0000 complete [ 517.106452] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 517.112344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 517.123106] Lustre: Skipped 2 previous similar messages [ 522.149790] Lustre: server umount lustre-MDT0000 complete [ 522.213150] LustreError: 15092:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 526.406569] LustreError: 15877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440366 with bad export cookie 18255456302599899786 [ 526.419417] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 526.749655] Lustre: server umount lustre-MDT0001 complete [ 531.433200] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 08:59:30 (1788440370) [ 534.221651] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 536.110455] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 08:59:35 (1788440375) [ 543.634901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 544.246424] LustreError: 19459:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 544.268457] LustreError: 19459:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 544.398204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 549.126681] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 549.857524] LustreError: 19460:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 558.115034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 562.985614] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 571.225738] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 577.779403] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 585.304601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 594.009221] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 604.466545] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 613.349316] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 613.358508] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 613.366725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 613.371359] Lustre: Skipped 1 previous similar message [ 615.115558] Lustre: server umount lustre-OST0000 complete [ 620.001082] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 621.547352] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 621.575092] Lustre: Skipped 1 previous similar message [ 631.777506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 631.787299] Lustre: Skipped 6 previous similar messages [ 634.335128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 634.521216] Lustre: server umount lustre-MDT0000 complete [ 636.898367] LustreError: 20187:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 636.911642] LustreError: 20187:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 638.469383] LustreError: 19441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440478 with bad export cookie 18255456302599900668 [ 638.471726] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 638.484832] LustreError: 19441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 638.863993] Lustre: server umount lustre-MDT0001 complete [ 650.971901] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 654.806956] Key type lgssc unregistered [ 655.286921] LNet: 23004:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 655.298208] LNetError: 23004:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 655.318758] LNet: Removed LNI 192.168.201.127@tcp [ 656.290405] Key type .llcrypt unregistered [ 656.294533] Key type ._llcrypt unregistered [ 680.403647] Key type ._llcrypt registered [ 680.406421] Key type .llcrypt registered [ 680.562318] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 681.958120] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 682.008248] alg: No test for adler32 (adler32-zlib) [ 683.002557] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 683.214161] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 684.875574] Key type lgssc registered [ 686.185749] Lustre: Echo OBD driver; http://www.lustre.org/ [ 695.426169] loop: module loaded [ 695.567305] loop0: detected capacity change from 0 to 204800000 [ 695.843741] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 695.862263] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 696.268338] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 702.442607] loop0: detected capacity change from 0 to 204800000 [ 702.648265] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 702.655152] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 702.670693] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 707.882104] loop0: detected capacity change from 0 to 204800000 [ 713.354469] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 713.377960] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 714.769514] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 714.796934] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 714.868460] Lustre: lustre-MDT0000: new disk, initializing [ 714.988683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 715.006299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 719.270938] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 727.897234] loop1: detected capacity change from 0 to 204800000 [ 734.358374] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 734.621826] Lustre: lustre-OST07c6: new disk, initializing [ 734.625425] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 734.629327] Lustre: Skipped 1 previous similar message [ 734.633393] Lustre: 26191:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 734.699136] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 739.948265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 739.956097] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 740.001124] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 741.303867] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 746.692985] Lustre: 25180:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 746.699228] Lustre: 25180:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 746.704597] Lustre: 25180:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 746.709140] Lustre: 25180:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 746.714152] Lustre: 25180:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 746.723886] Lustre: 25180:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 764.300495] Lustre: 25179:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 764.309769] Lustre: 25179:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 764.319221] Lustre: 25179:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 764.325584] Lustre: 25179:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 764.334350] Lustre: 25179:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 764.343750] Lustre: 25179:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 769.708490] Lustre: 25179:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 769.716785] Lustre: 25179:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 769.723256] Lustre: 25179:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 769.731028] Lustre: 25179:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 769.734828] Lustre: 25179:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 769.740965] Lustre: 25179:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 769.746116] Lustre: 25179:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 769.752806] Lustre: 25179:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 769.756865] Lustre: 25179:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 769.765589] Lustre: 25179:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 769.771406] Lustre: 25179:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 769.781115] Lustre: 25179:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 772.950825] Lustre: 25181:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 772.958651] Lustre: 25181:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 772.964150] Lustre: 25181:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 772.968018] Lustre: 25181:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 772.971645] Lustre: 25181:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 772.975943] Lustre: 25181:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 772.980870] Lustre: 25181:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 772.985381] Lustre: 25181:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 772.989918] Lustre: 25181:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 772.994853] Lustre: 25181:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 773.003117] Lustre: 25181:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 773.006830] Lustre: 25181:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 777.882719] Lustre: 25180:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 777.902243] Lustre: 25180:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 777.918542] Lustre: 25180:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 777.929570] Lustre: 25180:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 777.936950] Lustre: 25180:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 777.944580] Lustre: 25180:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 777.958564] Lustre: 25180:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 777.966864] Lustre: 25180:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 777.975816] Lustre: 25180:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 777.981972] Lustre: 25180:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 778.001362] Lustre: 25180:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 778.015301] Lustre: 25180:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 789.240119] Lustre: 25180:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 789.245427] Lustre: 25180:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 789.251044] Lustre: 25180:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 789.258442] Lustre: 25180:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 789.265330] Lustre: 25180:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 789.271154] Lustre: 25180:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 789.277801] Lustre: 25180:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 789.283520] Lustre: 25180:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 789.288958] Lustre: 25180:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 789.293967] Lustre: 25180:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 789.299223] Lustre: 25180:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 789.313989] Lustre: 25180:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 842.218906] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 842.231199] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 845.344767] Lustre: server umount lustre-OST07c6 complete [ 855.616935] Lustre: server umount lustre-MDT0000 complete [ 872.495709] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 874.982798] Key type lgssc unregistered [ 875.226029] LNet: 28746:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 875.240741] LNetError: 28746:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 875.275809] LNet: Removed LNI 192.168.201.127@tcp [ 876.306140] Key type .llcrypt unregistered [ 876.308694] Key type ._llcrypt unregistered [ 895.955440] Key type ._llcrypt registered [ 895.959194] Key type .llcrypt registered [ 896.076793] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 897.441894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 897.695665] alg: No test for adler32 (adler32-zlib) [ 898.763170] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 899.120699] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 900.819261] Key type lgssc registered [ 902.166424] Lustre: Echo OBD driver; http://www.lustre.org/ [ 910.035281] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 910.051796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 911.511404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 917.074863] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 921.057912] LustreError: 30016:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 921.080720] LustreError: 30016:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 926.179335] LustreError: 30015:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 926.647915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 927.010022] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 931.533920] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 938.642744] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 943.907564] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 950.486421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 950.876941] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 956.050640] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 957.174774] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 966.446370] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 983.009988] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 983.019364] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 983.033519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 987.120750] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 987.121696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 988.674732] Lustre: server umount lustre-OST0000 complete [ 993.763634] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 993.776107] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 993.801598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 993.804383] Lustre: Skipped 1 previous similar message [ 997.346729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 997.358597] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 998.911452] Lustre: server umount lustre-MDT0000 complete [ 1002.467488] LustreError: 30016:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1002.487871] LustreError: 30016:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1002.705557] LustreError: 29996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440842 with bad export cookie 14595963130466027261 [ 1002.711428] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1002.712781] LustreError: 29996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1002.935611] Lustre: server umount lustre-MDT0001 complete [ 1014.514341] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 1016.930210] Key type lgssc unregistered [ 1017.176859] LNet: 33750:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1017.181947] LNetError: 33750:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1018.236652] LNet: Removed LNI 192.168.201.127@tcp [ 1019.060115] Key type .llcrypt unregistered [ 1019.062158] Key type ._llcrypt unregistered [ 1039.424417] Key type ._llcrypt registered [ 1039.428850] Key type .llcrypt registered [ 1039.573810] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 1041.098180] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1041.138334] alg: No test for adler32 (adler32-zlib) [ 1042.217659] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1042.526775] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 1044.223205] Key type lgssc registered [ 1045.188992] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1052.570295] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1052.593771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1054.194183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1058.633606] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 1064.421511] LustreError: 35018:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1064.456235] LustreError: 35018:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1068.092056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1068.542745] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1073.430623] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 1083.549350] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1091.708688] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1100.610192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1100.939040] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1107.908855] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 1116.217201] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1125.549766] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1140.198727] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1140.206372] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1140.220279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1141.217264] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1141.225126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1141.235200] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1141.258703] Lustre: Skipped 1 previous similar message [ 1143.961691] Lustre: server umount lustre-OST0000 complete [ 1150.433650] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1150.438289] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1150.457042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1153.882596] Lustre: server umount lustre-MDT0000 complete [ 1156.594505] LustreError: 37509:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1156.632439] LustreError: 37509:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1158.315642] LustreError: 34999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440998 with bad export cookie 1384038468605228185 [ 1158.334951] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1158.912616] Lustre: server umount lustre-MDT0001 complete [ 1170.510783] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 1173.553724] Key type lgssc unregistered [ 1173.856636] LNet: 38577:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1173.864732] LNetError: 38577:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1173.953345] LNet: Removed LNI 192.168.201.127@tcp [ 1175.060058] Key type .llcrypt unregistered [ 1175.068957] Key type ._llcrypt unregistered [ 1201.662735] Key type ._llcrypt registered [ 1201.669865] Key type .llcrypt registered [ 1201.865249] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 1203.430471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1203.492053] alg: No test for adler32 (adler32-zlib) [ 1204.514445] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1204.813416] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 1206.583625] Key type lgssc registered [ 1208.025770] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1215.667575] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1215.722046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1217.591485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1223.768727] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 1227.746482] LustreError: 39993:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1227.755710] LustreError: 39993:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1232.866923] LustreError: 39994:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1235.284996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1235.686229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1240.614621] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 1248.938731] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1254.533111] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1261.505487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1261.873932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1262.923632] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1268.821635] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 1275.481410] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1285.605757] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1285.616875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1285.642204] Lustre: Skipped 2 previous similar messages [ 1285.653174] Lustre: Skipped 1 previous similar message [ 1287.718899] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.27@tcp (stopping) [ 1287.726470] Lustre: Skipped 1 previous similar message [ 1290.722214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1290.732080] Lustre: Skipped 2 previous similar messages [ 1291.035550] Lustre: server umount lustre-MDT0000 complete [ 1292.829081] LustreError: 39990:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1294.453587] LustreError: 39973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441134 with bad export cookie 4628840866819299441 [ 1294.462566] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1294.469050] LustreError: 39973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1295.841326] LustreError: 39989:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1295.842332] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1295.857272] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1299.874228] LustreError: 39988:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1299.874298] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1299.891894] LustreError: 39988:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1299.924336] Lustre: Skipped 1 previous similar message [ 1300.848970] Lustre: server umount lustre-MDT0001 complete [ 1322.381289] Lustre: server umount lustre-OST0000 complete [ 1342.451550] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 1346.867974] Key type lgssc unregistered [ 1347.280037] LNet: 43855:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1347.291874] LNetError: 43855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1347.326928] LNet: Removed LNI 192.168.201.127@tcp [ 1348.637693] Key type .llcrypt unregistered [ 1348.640666] Key type ._llcrypt unregistered [ 1378.163707] Key type ._llcrypt registered [ 1378.166398] Key type .llcrypt registered [ 1378.262147] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 1379.463709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1379.858054] alg: No test for adler32 (adler32-zlib) [ 1380.926273] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1381.257883] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 1382.983204] Key type lgssc registered [ 1384.528092] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1395.758238] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1395.799451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1397.575430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1402.565437] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 1406.956092] LustreError: 45270:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1406.972922] LustreError: 45270:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1412.047204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1412.073435] LustreError: 45271:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1412.744290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1417.499528] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 1425.475923] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1432.422496] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1439.524459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1439.906281] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1441.985019] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1446.481859] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 1453.967040] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1466.926778] Lustre: lustre-OST0000: Not available for connect from 192.168.201.27@tcp (stopping) [ 1467.874158] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1467.879910] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1467.887847] LustreError: Skipped 1 previous similar message [ 1467.888516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1467.923434] Lustre: Skipped 1 previous similar message [ 1470.178439] Lustre: server umount lustre-OST0000 complete [ 1489.377376] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1489.383889] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1489.405048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1489.411533] Lustre: Skipped 1 previous similar message [ 1493.472570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1493.474386] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1495.233078] Lustre: server umount lustre-MDT0000 complete [ 1498.594509] LustreError: 45266:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1498.627773] LustreError: 45266:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1499.520727] LustreError: 45252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441339 with bad export cookie 13064967565261383627 [ 1499.522778] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1499.532882] LustreError: 45252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1506.015593] Lustre: server umount lustre-MDT0001 complete [ 1519.895394] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 1522.797331] Key type lgssc unregistered [ 1523.050658] LNet: 48977:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1523.057164] LNetError: 48977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1524.076873] LNet: Removed LNI 192.168.201.127@tcp [ 1524.771811] Key type .llcrypt unregistered [ 1524.775610] Key type ._llcrypt unregistered [ 1549.942387] Key type ._llcrypt registered [ 1549.947194] Key type .llcrypt registered [ 1550.144283] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 1551.093824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1551.225203] alg: No test for adler32 (adler32-zlib) [ 1552.279039] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1552.490503] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 1554.151198] Key type lgssc registered [ 1555.495787] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1563.490845] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1563.531440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1565.046911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1569.913496] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 1575.400839] LustreError: 50394:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1575.409363] LustreError: 50394:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1578.415444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1578.844668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1583.309519] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 1591.721421] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1598.692481] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1606.130851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1606.414462] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1610.548509] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1612.918888] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 1622.207129] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1629.602362] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1634.020543] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1647.496110] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-03 9h18m05s [ 1649.806202] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1654.767337] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1654.786694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1654.798715] Lustre: Skipped 1 previous similar message [ 1654.827506] Lustre: Skipped 2 previous similar messages [ 1655.843739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1656.911689] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.27@tcp (stopping) [ 1656.933252] Lustre: Skipped 3 previous similar messages [ 1659.124801] Lustre: server umount lustre-MDT0000 complete [ 1661.949736] LustreError: 51432:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1661.977808] LustreError: 51432:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1662.986620] LustreError: 50390:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1663.862991] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1668.073658] LustreError: 50389:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1668.091979] LustreError: 50389:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1672.456478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1672.688895] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1673.045396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1673.112902] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1674.116641] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1678.068839] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 1681.673766] Lustre: DEBUG MARKER: Wait for df (36709) ... [ 1683.649599] Lustre: DEBUG MARKER: done [ 1688.545936] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1688.556201] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1688.584927] Lustre: Skipped 1 previous similar message [ 1688.595831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1689.573672] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1694.732409] Lustre: server umount lustre-OST0000 complete [ 1699.811349] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1699.817824] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1699.837721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1699.857546] Lustre: Skipped 1 previous similar message [ 1699.873919] Lustre: Skipped 3 previous similar messages [ 1705.411428] Lustre: server umount lustre-MDT0000 complete [ 1706.981114] LustreError: 51432:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1707.012529] LustreError: 51432:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1709.766793] LustreError: 53024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441549 with bad export cookie 10372917557428284201 [ 1709.774786] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1709.786090] LustreError: 53024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1710.183281] Lustre: server umount lustre-MDT0001 complete [ 1724.092187] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 1726.686949] Key type lgssc unregistered [ 1726.908841] LNet: 55323:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1726.912923] LNetError: 55323:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1726.927788] LNet: Removed LNI 192.168.201.127@tcp [ 1727.706347] Key type .llcrypt unregistered [ 1727.710406] Key type ._llcrypt unregistered [ 1773.818223] Key type ._llcrypt registered [ 1773.821636] Key type .llcrypt registered [ 1773.936976] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 1775.292909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1775.492547] alg: No test for adler32 (adler32-zlib) [ 1776.682312] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1776.986440] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 1778.687224] Key type lgssc registered [ 1779.805816] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1788.224179] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1788.242182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1789.543830] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1789.575403] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1789.713977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1794.261598] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 1803.078754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1803.178131] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1803.231466] Lustre: 57531:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1803.481298] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1808.014577] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 1815.579630] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1821.901861] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1829.291140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1829.507813] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1829.715053] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1836.339707] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 1836.779535] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1844.809951] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1853.204776] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-03 9h21m32s [ 1854.913265] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1864.471714] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1867.620398] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1867.674867] LustreError: 57536:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1887.775106] Lustre: lustre-MDT0000: Client b8ce296c-f5ba-4753-9513-d7a03e6ad609 (at 192.168.201.27@tcp) reconnecting [ 1907.767203] LustreError: 57536:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1910.429629] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1913.556236] Lustre: DEBUG MARKER: done [ 1924.066733] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1924.070519] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1924.074148] LustreError: Skipped 1 previous similar message [ 1924.078474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1924.099485] Lustre: Skipped 1 previous similar message [ 1926.503128] Lustre: server umount lustre-OST0000 complete [ 1931.748155] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1931.757978] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1931.766648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1931.770537] Lustre: Skipped 1 previous similar message [ 1932.835699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1932.838395] Lustre: Skipped 2 previous similar messages [ 1935.908615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1935.922204] Lustre: Skipped 1 previous similar message [ 1937.334403] Lustre: server umount lustre-MDT0000 complete [ 1941.144978] LustreError: 59392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441781 with bad export cookie 2512028581839084054 [ 1941.167106] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1941.169442] LustreError: 59392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1941.556614] Lustre: server umount lustre-MDT0001 complete [ 1954.301460] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 1956.807318] Key type lgssc unregistered [ 1957.141793] LNet: 61867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1957.150194] LNetError: 61867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1957.203438] LNet: Removed LNI 192.168.201.127@tcp [ 1958.286588] Key type .llcrypt unregistered [ 1958.289068] Key type ._llcrypt unregistered [ 2010.564792] Key type ._llcrypt registered [ 2010.567727] Key type .llcrypt registered [ 2010.733975] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 2011.996207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2012.117196] alg: No test for adler32 (adler32-zlib) [ 2013.250652] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2013.790126] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 2015.623192] Key type lgssc registered [ 2017.017568] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2021.869787] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2021.883043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2023.270423] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2023.291747] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2023.473791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2024.465741] Lustre: Failing over lustre-MDT0000 [ 2024.717778] Lustre: server umount lustre-MDT0000 complete [ 2032.403571] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 09:24:31 (1788441871) [ 2039.599158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2039.847911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2044.979787] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 2054.026288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2054.183763] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2054.208823] Lustre: 64666:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2054.477992] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2058.973827] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 2066.997255] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2073.158940] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2080.655723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2080.929042] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2081.150354] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2087.778058] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 2088.156720] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2095.725033] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2101.871646] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2110.947250] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2110.959472] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2110.980586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2114.048363] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2114.051577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2114.069641] Lustre: Skipped 1 previous similar message [ 2114.077511] Lustre: Skipped 1 previous similar message [ 2115.749760] Lustre: server umount lustre-MDT0000 complete [ 2119.986602] LustreError: 65403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441960 with bad export cookie 2373502365306771485 [ 2119.989046] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2120.001920] LustreError: 65403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2120.458091] Lustre: server umount lustre-MDT0001 complete [ 2124.507688] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2130.743768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2140.640039] Lustre: 63589:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788441964/real 1788441964] req@00000000684a2edf x1875317204494336/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788441980 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2140.665729] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2141.119110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2145.759313] Lustre: 63588:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788441969/real 1788441969] req@000000003a5e3477 x1875317204494464/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788441985 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2146.033635] LustreError: 66516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2146.048636] LustreError: 66516:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2146.122751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2146.215608] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2147.168861] Lustre: 63590:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788441970/real 1788441970] req@000000006b5d0b74 x1875317204494720/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788441986 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2149.154855] LustreError: 66517:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2150.179125] LustreError: 66516:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2150.199918] LustreError: 66516:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2151.200377] Lustre: 63591:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788441975/real 1788441975] req@0000000064d62ebf x1875317204494848/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788441991 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2151.617993] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 2155.491450] LustreError: 66517:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2155.526559] LustreError: 66517:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2156.512477] Lustre: 63588:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788441980/real 1788441980] req@000000006211beaa x1875317204494976/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788441996 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2160.284948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2160.695346] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2165.443062] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 2165.749413] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2168.890444] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2173.092644] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2179.138484] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2189.287956] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2189.292108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2189.311194] Lustre: Skipped 1 previous similar message [ 2189.316984] Lustre: Skipped 1 previous similar message [ 2193.019850] Lustre: server umount lustre-MDT0000 complete [ 2194.401179] LustreError: 68432:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2194.430844] LustreError: 68432:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2197.131893] LustreError: 64660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442037 with bad export cookie 2373502365306773760 [ 2197.136927] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2197.142533] LustreError: 64660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2197.462834] Lustre: server umount lustre-MDT0001 complete [ 2204.035679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2214.270117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2215.649271] Lustre: 63591:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788442039/real 1788442039] req@00000000075c34c8 x1875317204522624/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788442055 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2215.680566] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2215.703134] Lustre: Skipped 1 previous similar message [ 2221.926813] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x20f05fd36da8e663 [ 2221.951730] Lustre: MGC192.168.201.127@tcp: Connection restored to 0@lo (at 0@lo) [ 2222.305977] LustreError: 66517:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2222.401571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2222.462893] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2227.823364] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 2239.121992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2244.069909] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2244.712516] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 2253.975080] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2260.962918] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2269.666463] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2269.668095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2269.684963] Lustre: Skipped 1 previous similar message [ 2269.694404] Lustre: Skipped 2 previous similar messages [ 2274.787550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2274.801840] Lustre: Skipped 2 previous similar messages [ 2275.266133] Lustre: server umount lustre-MDT0000 complete [ 2279.207180] LustreError: 64658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442119 with bad export cookie 2373502365306775139 [ 2279.219456] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2279.229462] LustreError: 64658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2279.642974] Lustre: server umount lustre-MDT0001 complete [ 2286.896876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2291.180886] Lustre: DEBUG MARKER: files compared the same [ 2304.560990] Lustre: server umount lustre-OST0000 complete [ 2326.089617] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 2329.183334] Key type lgssc unregistered [ 2329.535768] LNet: 74395:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2329.543546] LNetError: 74395:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2329.571444] LNet: Removed LNI 192.168.201.127@tcp [ 2330.473995] Key type .llcrypt unregistered [ 2330.476545] Key type ._llcrypt unregistered [ 2355.435203] Key type ._llcrypt registered [ 2355.438961] Key type .llcrypt registered [ 2355.614806] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 2356.949235] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2357.091700] alg: No test for adler32 (adler32-zlib) [ 2358.266637] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2358.519540] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 2360.207185] Key type lgssc registered [ 2361.513571] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2369.169693] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2369.181615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2370.832432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2375.719533] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug +malloc all [ 2380.256391] LustreError: 75807:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2380.286673] LustreError: 75807:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2385.007097] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2385.242154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2389.710841] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug +malloc all [ 2397.953295] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2404.587558] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2411.600909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2411.970674] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2415.016118] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2419.044465] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug +malloc all [ 2426.352323] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2440.671807] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2440.672301] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2440.674121] LustreError: Skipped 1 previous similar message [ 2440.682894] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2440.686825] Lustre: Skipped 1 previous similar message [ 2440.710649] Lustre: Skipped 1 previous similar message [ 2444.870530] Lustre: server umount lustre-OST0000 complete [ 2450.920479] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2450.924049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2450.930250] Lustre: Skipped 1 previous similar message [ 2450.942676] Lustre: Skipped 1 previous similar message [ 2454.935242] Lustre: server umount lustre-MDT0000 complete [ 2456.035154] LustreError: 75801:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2456.056024] LustreError: 75801:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2458.934672] LustreError: 75785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442299 with bad export cookie 1027262232676963963 [ 2458.944129] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2458.946258] LustreError: 75785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2459.370680] Lustre: server umount lustre-MDT0001 complete [ 2470.029430] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 2472.544806] Key type lgssc unregistered [ 2472.824640] LNet: 79358:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2472.829983] LNetError: 79358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2473.892864] LNet: Removed LNI 192.168.201.127@tcp [ 2474.716253] Key type .llcrypt unregistered [ 2474.718208] Key type ._llcrypt unregistered [ 2505.105200] Key type ._llcrypt registered [ 2505.110156] Key type .llcrypt registered [ 2505.172656] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 2506.420097] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2506.514344] alg: No test for adler32 (adler32-zlib) [ 2507.623509] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2507.978839] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 2509.623423] Key type lgssc registered [ 2510.767219] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2519.570236] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2519.614432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2535.199707] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2540.319507] LustreError: 80748:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.127@tcp: failed processing log, type 4: rc = -110 [ 2566.943194] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2572.982344] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 2580.093284] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2588.387113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2588.525973] Lustre: *** cfs_fail_loc=706, val=0*** [ 2588.539636] LustreError: 80082:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000002e24c879 x1875317723498752/t0(0) o250->MGC192.168.201.127@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2588.912515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2588.982464] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2593.416697] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 2594.284022] LustreError: 80773:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2594.304877] LustreError: 80773:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2599.411077] LustreError: 80774:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2601.506815] LustreError: 81835:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2601.537150] LustreError: 81835:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2602.333094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2602.615869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2608.022337] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 2617.722526] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2624.466440] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2628.072957] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2628.082047] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2628.101597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2628.594637] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2628.603521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2633.705137] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2638.816481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2638.818352] Lustre: Skipped 1 previous similar message [ 2641.375735] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2641.650225] Lustre: server umount lustre-OST0000 complete [ 2648.544441] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2648.554885] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2648.595395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2648.605545] Lustre: Skipped 1 previous similar message [ 2659.296927] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2659.297953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2659.312832] Lustre: Skipped 5 previous similar messages [ 2659.425728] Lustre: server umount lustre-MDT0000 complete [ 2662.686469] LustreError: 80754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442502 with bad export cookie 15278280731876921694 [ 2662.698307] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2662.707594] LustreError: 80754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2662.992351] Lustre: server umount lustre-MDT0001 complete [ 2675.886801] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 2678.899451] Key type lgssc unregistered [ 2679.208777] LNet: 84364:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2679.225289] LNetError: 84364:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2679.254380] LNet: Removed LNI 192.168.201.127@tcp [ 2680.285070] Key type .llcrypt unregistered [ 2680.291976] Key type ._llcrypt unregistered [ 2705.179343] Key type ._llcrypt registered [ 2705.181626] Key type .llcrypt registered [ 2705.300675] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 2706.298447] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2706.397270] alg: No test for adler32 (adler32-zlib) [ 2707.514594] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2707.913954] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 2709.631993] Key type lgssc registered [ 2710.657284] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2717.386756] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2717.404417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2722.969902] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 2731.371113] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2732.107033] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2736.191902] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 2737.124689] LustreError: 86464:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2737.142089] LustreError: 86464:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2742.246842] LustreError: 86463:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2742.266113] LustreError: 86463:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2744.876197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2745.287989] LustreError: 86462:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2745.309262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2745.315827] LustreError: 86462:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2750.441049] LustreError: 86468:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2750.455557] LustreError: 86468:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2751.226410] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 2755.569374] LustreError: 86464:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2755.589667] LustreError: 86464:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2759.917925] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2759.928809] Lustre: MGS: Client 5c1c808a-e841-4f0f-90c2-59c8e78f6523 (at 0@lo) reconnecting [ 2759.943757] LustreError: 86478:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2760.157501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2760.284350] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2765.026382] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 2780.645897] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2780.656355] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2780.670902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2780.681214] Lustre: Skipped 1 previous similar message [ 2784.789927] Lustre: server umount lustre-OST0000 complete [ 2790.881482] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2790.881679] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2790.887091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2790.931808] Lustre: Skipped 2 previous similar messages [ 2794.486976] Lustre: server umount lustre-MDT0000 complete [ 2796.002370] LustreError: 86468:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2796.021473] LustreError: 86468:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2798.457983] Lustre: server umount lustre-MDT0001 complete [ 2802.224628] Lustre: server umount MGS complete [ 2813.337746] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 09:37:32 (1788442652) [ 2849.642549] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 2859.975358] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 2899.985111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2905.201996] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 2914.997282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2915.226765] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2915.258422] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2915.342425] Lustre: lustre-MDT0001: new disk, initializing [ 2915.446040] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2915.457925] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2915.463219] Lustre: 93638:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2915.469540] LustreError: 93638:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2920.153725] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 2932.373065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2932.718818] Lustre: lustre-OST0000: new disk, initializing [ 2932.722560] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2932.729591] Lustre: 94404:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2932.806674] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2942.576364] Lustre: 94698:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000fdf8a31a x1875317933252096/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788442798 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2943.045668] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 2958.774377] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2963.424206] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2967.533581] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2967.552550] Lustre: MGS: Client 0bbc6012-d3d0-448d-999c-a3dc5a4c0d4e (at 0@lo) reconnecting [ 2967.578563] LustreError: 93624:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2967.662410] Lustre: 95452:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2967.724873] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2967.859086] Lustre: lustre-MDT0000: new disk, initializing [ 2968.064630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2968.111072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2969.184688] Lustre: 95476:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000072ebc29e x1875317933260160/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788442825 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2971.697295] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 2974.239769] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2974.389498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2974.412229] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2974.455044] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 2978.283848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 2978.293681] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 3004.384719] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3004.391695] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3004.416802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3004.427029] Lustre: Skipped 1 previous similar message [ 3009.178687] Lustre: server umount lustre-OST0000 complete [ 3015.651075] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3015.660989] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3015.668378] LustreError: Skipped 1 previous similar message [ 3015.683173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3015.705270] Lustre: Skipped 2 previous similar messages [ 3015.726149] Lustre: Skipped 3 previous similar messages [ 3020.087865] Lustre: server umount lustre-MDT0000 complete [ 3020.782539] LustreError: 93614:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3020.810276] LustreError: 93614:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3030.954684] Lustre: server umount lustre-MDT0001 complete [ 3034.932223] Lustre: server umount MGS complete [ 3048.072808] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 09:41:27 (1788442887) [ 3069.888821] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 3072.666625] Key type lgssc unregistered [ 3072.988743] LNet: 97983:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3073.006485] LNetError: 97983:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3074.027787] LNet: Removed LNI 192.168.201.127@tcp [ 3075.111350] Key type .llcrypt unregistered [ 3075.113516] Key type ._llcrypt unregistered [ 3093.963799] Key type ._llcrypt registered [ 3093.968455] Key type .llcrypt registered [ 3094.111968] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 3095.636599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3095.665048] alg: No test for adler32 (adler32-zlib) [ 3096.937893] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3097.350348] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 3099.143191] Key type lgssc registered [ 3100.256333] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3107.626702] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3107.641103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3109.250751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3113.907298] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 3119.590126] LustreError: 99204:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3119.620090] LustreError: 99204:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3123.667652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3124.096756] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3129.689573] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 3140.610839] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3149.123268] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3159.741864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3160.237265] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3168.755615] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 3178.561966] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3179.704878] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:33) [ 3190.241678] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3190.255856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3190.283422] Lustre: Skipped 1 previous similar message [ 3190.298080] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3190.303107] Lustre: Skipped 1 previous similar message [ 3194.791428] Lustre: server umount lustre-OST0000 complete [ 3200.481989] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3200.484523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3200.505441] Lustre: Skipped 1 previous similar message [ 3200.510088] Lustre: Skipped 1 previous similar message [ 3205.487854] Lustre: server umount lustre-MDT0000 complete [ 3205.603306] LustreError: 99205:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3205.617954] LustreError: 99205:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3208.871938] LustreError: 99184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443049 with bad export cookie 909511004242611729 [ 3208.873407] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3208.889174] LustreError: 99184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3209.399322] Lustre: server umount lustre-MDT0001 complete [ 3222.258538] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 3225.290471] Key type lgssc unregistered [ 3225.618346] LNet: 102743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3225.632092] LNetError: 102743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3226.664416] LNet: Removed LNI 192.168.201.127@tcp [ 3227.369764] Key type .llcrypt unregistered [ 3227.371161] Key type ._llcrypt unregistered [ 3245.510225] Key type ._llcrypt registered [ 3245.520290] Key type .llcrypt registered [ 3245.635761] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 3247.024404] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3247.105588] alg: No test for adler32 (adler32-zlib) [ 3248.186842] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3248.395975] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 3250.090292] Key type lgssc registered [ 3251.137516] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3257.957850] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3257.957948] LustreError: 103822:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 3257.962159] Lustre: Skipped 1 previous similar message [ 3257.962184] LustreError: 103823:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 3257.984366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3257.991109] LustreError: 103822:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 3258.040850] LustreError: 103822:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3258.055481] LustreError: 103822:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3259.648862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3267.570169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3268.007952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3273.518653] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 3280.461845] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing lsmod [ 3284.044502] LustreError: 105027:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 3284.062785] LustreError: 105028:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 3284.068312] LustreError: 105027:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4984 [ 3284.079496] LustreError: 105027:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3284.082508] LustreError: 105027:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3284.094493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3284.338505] LustreError: 105038:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3284.381131] LustreError: 105038:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3284.633258] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3288.544280] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3288.552558] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3288.573902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3290.085627] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3290.087789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3290.108710] Lustre: Skipped 1 previous similar message [ 3290.128332] Lustre: Skipped 2 previous similar messages [ 3293.236259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3293.249386] Lustre: Skipped 2 previous similar messages [ 3294.038918] Lustre: server umount lustre-MDT0000 complete [ 3298.276447] LustreError: 103861:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3298.294956] LustreError: 103861:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3298.351803] LustreError: 103835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443138 with bad export cookie 8618666980198920724 [ 3298.371085] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3299.423324] LustreError: 104291:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000004094b2ee x1875318499464704/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3299.717602] Lustre: server umount lustre-MDT0001 complete [ 3315.091739] Lustre: server umount lustre-OST0000 complete [ 3324.151411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3324.787461] LustreError: 106094:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3324.917845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3330.035319] LustreError: 106095:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3330.353706] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 3335.141589] LustreError: 106094:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3340.438790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3340.890305] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3346.154138] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 3354.384051] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3361.599642] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3369.951233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3370.352976] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3371.389638] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:65) [ 3378.488153] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 3387.750868] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3396.068862] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3396.071887] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3396.072231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3396.083174] Lustre: Skipped 1 previous similar message [ 3401.186587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3401.198870] Lustre: Skipped 2 previous similar messages [ 3401.878426] Lustre: server umount lustre-OST0000 complete [ 3406.308607] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3411.429377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3411.442807] Lustre: Skipped 3 previous similar messages [ 3412.190311] Lustre: server umount lustre-MDT0000 complete [ 3415.767871] LustreError: 106076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443255 with bad export cookie 8618666980198921424 [ 3415.771969] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3415.779950] LustreError: 106076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3416.240886] Lustre: server umount lustre-MDT0001 complete [ 3427.601653] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 3430.662515] Key type lgssc unregistered [ 3430.946197] LNet: 109627:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3430.963384] LNetError: 109627:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3432.009076] LNet: Removed LNI 192.168.201.127@tcp [ 3433.003189] Key type .llcrypt unregistered [ 3433.005540] Key type ._llcrypt unregistered [ 3458.763306] Key type ._llcrypt registered [ 3458.765930] Key type .llcrypt registered [ 3458.858415] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 3460.008419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3460.045223] alg: No test for adler32 (adler32-zlib) [ 3461.167335] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3461.423925] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 3463.135181] Key type lgssc registered [ 3464.296384] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3472.938890] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3472.966728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3474.490788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3479.050979] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 3484.645722] LustreError: 111043:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3484.667716] LustreError: 111043:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3488.740082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3489.259141] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3494.696774] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 3503.674806] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3510.811656] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3517.871744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3518.240729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3519.292263] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:67 to 0x240000400:97) [ 3524.608331] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 3532.763976] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3539.496362] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3544.711032] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3550.176823] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3550.179138] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3550.189601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3550.205838] Lustre: Skipped 1 previous similar message [ 3550.222723] Lustre: Skipped 1 previous similar message [ 3555.299048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3555.305094] Lustre: Skipped 1 previous similar message [ 3555.840364] Lustre: server umount lustre-OST0000 complete [ 3560.417556] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3560.434669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3560.440553] Lustre: Skipped 1 previous similar message [ 3565.543488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3565.550409] Lustre: Skipped 1 previous similar message [ 3567.117784] Lustre: server umount lustre-MDT0000 complete [ 3570.657400] LustreError: 111043:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3570.683131] LustreError: 111043:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3572.577428] LustreError: 111776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443412 with bad export cookie 14906696179031752312 [ 3572.581114] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3572.587211] LustreError: 111776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3573.582881] Lustre: server umount lustre-MDT0001 complete [ 3589.428913] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 3593.546663] Key type lgssc unregistered [ 3593.913128] LNet: 114679:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3593.916351] LNetError: 114679:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3593.947172] LNet: Removed LNI 192.168.201.127@tcp [ 3595.349487] Key type .llcrypt unregistered [ 3595.352250] Key type ._llcrypt unregistered [ 3616.681384] Key type ._llcrypt registered [ 3616.684644] Key type .llcrypt registered [ 3616.788711] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 3617.796993] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3617.918108] alg: No test for adler32 (adler32-zlib) [ 3619.110729] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3619.326806] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 3621.023171] Key type lgssc registered [ 3622.274435] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3630.432778] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3630.451758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3631.969477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3637.532571] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 3642.339640] LustreError: 115903:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3642.361293] LustreError: 115903:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3647.459739] LustreError: 115904:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3647.926865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3648.479627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3654.300397] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 3663.937582] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3671.141414] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3679.913048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3680.274541] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3687.139239] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 3694.666545] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3702.848598] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 3718.146803] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3718.171926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3718.176651] Lustre: Skipped 1 previous similar message [ 3718.178896] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3718.199434] Lustre: Skipped 1 previous similar message [ 3721.944289] Lustre: server umount lustre-OST0000 complete [ 3728.362066] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3728.373501] Lustre: Skipped 1 previous similar message [ 3728.379599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3728.394952] Lustre: Skipped 1 previous similar message [ 3733.037981] Lustre: server umount lustre-MDT0000 complete [ 3733.480323] LustreError: 115898:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3733.493181] LustreError: 115898:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3738.497375] LustreError: 115884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443578 with bad export cookie 14915577650306485727 [ 3738.509977] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3738.510803] LustreError: 115884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3739.147524] Lustre: server umount lustre-MDT0001 complete [ 3754.393134] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 3757.964837] Key type lgssc unregistered [ 3758.456780] LNet: 119460:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3758.469629] LNetError: 119460:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3759.532170] LNet: Removed LNI 192.168.201.127@tcp [ 3760.446803] Key type .llcrypt unregistered [ 3760.450693] Key type ._llcrypt unregistered [ 3790.484322] Key type ._llcrypt registered [ 3790.486575] Key type .llcrypt registered [ 3790.625248] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 3792.256710] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3792.323762] alg: No test for adler32 (adler32-zlib) [ 3793.466550] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3793.740879] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 3795.456988] Key type lgssc registered [ 3796.749514] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3805.969399] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3805.995372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3807.549773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3812.928140] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 3817.001337] LustreError: 120876:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3817.036323] LustreError: 120876:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3822.053163] LustreError: 120877:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3822.310395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3822.800182] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3826.708288] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 3835.746665] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3842.830526] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3849.904470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3850.266222] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3852.350152] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:131 to 0x240000400:161) [ 3857.669903] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 3866.868271] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3875.189202] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3878.912688] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3878.920335] Lustre: Skipped 2 previous similar messages [ 3882.198628] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3882.252966] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3884.474915] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3884.479988] Lustre: Skipped 2 previous similar messages [ 3890.335802] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3890.339862] Lustre: Skipped 1 previous similar message [ 3896.865698] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.27@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3896.872785] Lustre: Skipped 2 previous similar messages [ 3898.069451] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.27@tcp 192.168.0.[2,10]@tcp [ 3908.838371] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.27/24@tcp in log lustre-MDT0000 [ 3908.842648] Lustre: Skipped 2 previous similar messages [ 3918.388605] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.27/24@tcp [ 3918.396843] Lustre: Skipped 1 previous similar message [ 3925.993239] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3925.994268] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3926.002630] LustreError: Skipped 1 previous similar message [ 3926.037936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3927.515200] Lustre: server umount lustre-OST0000 complete [ 3934.692925] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3934.719135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3934.719606] Lustre: Skipped 1 previous similar message [ 3934.736698] Lustre: Skipped 1 previous similar message [ 3937.988683] Lustre: server umount lustre-MDT0000 complete [ 3939.824704] LustreError: 120876:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3939.854937] LustreError: 120876:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3941.985408] LustreError: 121610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443782 with bad export cookie 614507512162793161 [ 3941.998911] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3952.867986] Lustre: server umount lustre-MDT0001 complete [ 3965.104169] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 3967.815420] Key type lgssc unregistered [ 3968.111700] LNet: 125871:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3968.122301] LNetError: 125871:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3968.194146] LNet: Removed LNI 192.168.201.127@tcp [ 3969.214339] Key type .llcrypt unregistered [ 3969.216608] Key type ._llcrypt unregistered [ 3998.294300] Key type ._llcrypt registered [ 3998.296340] Key type .llcrypt registered [ 3998.500317] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 4000.117830] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4000.145177] alg: No test for adler32 (adler32-zlib) [ 4001.292831] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 4001.541449] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 4003.311576] Key type lgssc registered [ 4004.569520] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4011.898428] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4011.910421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4013.290652] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4013.306280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4013.317215] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.27/24@tcp [ 4017.729649] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 4023.776503] LustreError: 127428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4023.794493] LustreError: 127428:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4027.305635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4027.593346] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4027.607189] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.27/24@tcp [ 4027.643100] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4032.228436] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 4039.612422] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4045.891513] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4052.524508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4052.931766] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4058.052151] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:169 to 0x240000400:193) [ 4059.064912] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 4066.300467] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4081.125836] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4081.136359] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4081.156398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4082.666420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4082.668417] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4082.692737] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4086.093973] Lustre: server umount lustre-OST0000 complete [ 4092.912123] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4092.926105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4092.941819] Lustre: Skipped 2 previous similar messages [ 4096.504448] Lustre: server umount lustre-MDT0000 complete [ 4098.016995] LustreError: 127424:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4098.045901] LustreError: 127424:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4100.611867] LustreError: 127409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443940 with bad export cookie 5016079612202525716 [ 4100.618384] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4100.890725] Lustre: server umount lustre-MDT0001 complete [ 4115.321060] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 4119.875574] Key type lgssc unregistered [ 4120.421541] LNet: 131018:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4120.434516] LNetError: 131018:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4120.456484] LNet: Removed LNI 192.168.201.127@tcp [ 4122.088891] Key type .llcrypt unregistered [ 4122.092553] Key type ._llcrypt unregistered [ 4155.331622] Key type ._llcrypt registered [ 4155.333313] Key type .llcrypt registered [ 4155.532223] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 4175.007194] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 4176.101872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4176.141458] alg: No test for adler32 (adler32-zlib) [ 4177.301274] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 4177.504918] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 4179.199370] Key type lgssc registered [ 4180.885291] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4224.609801] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4224.701882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4226.190406] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4226.251491] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4226.403912] Lustre: lustre-MDT0000: new disk, initializing [ 4226.553144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4226.618329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4232.667168] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 4247.373151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4247.778693] Lustre: 134429:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4247.952552] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4247.957124] Lustre: Skipped 1 previous similar message [ 4248.119834] Lustre: lustre-MDT0001: new disk, initializing [ 4248.273288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4248.314829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4248.327270] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4253.245528] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 4264.159295] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4270.967656] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4279.375935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4279.675269] Lustre: lustre-OST0000: new disk, initializing [ 4279.678979] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4279.703947] Lustre: 136399:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4279.824055] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4286.944393] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 4288.566146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4288.578542] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4288.756434] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4298.706247] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4304.272698] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4304.544626] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4309.509000] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4309.830876] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4314.593128] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4314.594880] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4314.606396] LustreError: Skipped 1 previous similar message [ 4314.608096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4314.669226] Lustre: Skipped 1 previous similar message [ 4318.603424] Lustre: server umount lustre-OST0000 complete [ 4324.835798] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4324.854543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4324.857378] Lustre: Skipped 1 previous similar message [ 4324.869593] Lustre: Skipped 2 previous similar messages [ 4329.008824] Lustre: server umount lustre-MDT0000 complete [ 4333.021676] LustreError: 134422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444173 with bad export cookie 16448512883953396545 [ 4333.023886] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4333.030309] LustreError: 134422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4333.496077] Lustre: server umount lustre-MDT0001 complete [ 4343.333718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4344.070134] LustreError: 138619:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4344.097930] LustreError: 138619:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4344.200239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4349.414969] LustreError: 138620:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4350.731704] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 4354.536082] LustreError: 138619:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4359.661969] LustreError: 138620:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4360.841492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4361.091987] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4366.145231] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 4374.954767] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4381.783854] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4389.437634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4389.922622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4397.840062] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 4408.655456] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4413.901057] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4415.327557] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 4421.187296] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4421.428661] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4431.477146] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4431.696413] Lustre: lustre-OST0001: new disk, initializing [ 4431.699952] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4431.705688] Lustre: 141998:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4431.776550] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4437.886775] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 4441.739454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4441.756907] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4441.899546] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4446.562906] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4446.840306] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4452.818976] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4453.124591] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4461.499768] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff953d448ed800.ost_server_uuid 50 [ 4464.031819] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff953d448ed800.ost_server_uuid in IDLE state after 0 sec [ 4477.920503] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4477.925147] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4477.931756] LustreError: Skipped 1 previous similar message [ 4477.933705] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4477.968276] Lustre: Skipped 1 previous similar message [ 4482.218835] Lustre: server umount lustre-OST0001 complete [ 4483.054512] LustreError: 140440:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4483.087163] LustreError: 140440:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4488.160300] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4488.169985] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4488.171436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4488.185821] Lustre: Skipped 1 previous similar message [ 4488.198073] Lustre: Skipped 2 previous similar messages [ 4493.296845] LustreError: 142529:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4493.303231] LustreError: 142529:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4493.361277] Lustre: server umount lustre-OST0000 complete [ 4498.405222] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4498.417408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4498.427014] Lustre: Skipped 1 previous similar message [ 4504.474422] Lustre: server umount lustre-MDT0000 complete [ 4508.094095] LustreError: 142008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444348 with bad export cookie 16448512883953397427 [ 4508.100796] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4508.108761] LustreError: 142008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4508.650755] Lustre: server umount lustre-MDT0001 complete [ 4532.251549] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 4535.962368] Key type lgssc unregistered [ 4536.345643] LNet: 145003:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4536.354962] LNetError: 145003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4536.380950] LNet: Removed LNI 192.168.201.127@tcp [ 4537.636500] Key type .llcrypt unregistered [ 4537.641069] Key type ._llcrypt unregistered [ 4581.132337] Key type ._llcrypt registered [ 4581.137634] Key type .llcrypt registered [ 4581.240666] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 4602.880912] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 4604.290759] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4604.476346] alg: No test for adler32 (adler32-zlib) [ 4605.836857] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 4606.309449] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 4608.007110] Key type lgssc registered [ 4609.362101] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4649.486989] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4649.532509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4650.877611] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4650.925773] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4651.020512] Lustre: lustre-MDT0000: new disk, initializing [ 4651.164095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4651.190053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4656.471666] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 4668.088512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4668.224480] Lustre: 149157:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4668.274742] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4668.279952] Lustre: Skipped 1 previous similar message [ 4668.348637] Lustre: lustre-MDT0001: new disk, initializing [ 4668.429475] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4668.473611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4668.488143] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4673.628340] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 4683.885794] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4690.762927] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4698.473231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4698.889780] Lustre: lustre-OST0000: new disk, initializing [ 4698.893683] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4698.901067] Lustre: 151126:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4698.983063] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4706.426320] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 4708.025489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4708.048374] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4708.247448] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4717.476602] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4723.232554] Lustre: Failing over lustre-OST0000 [ 4723.459809] Lustre: server umount lustre-OST0000 complete [ 4739.046929] Lustre: 147380:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788444563/real 1788444563] req@00000000fae39941 x1875319923443456/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788444579 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4739.049446] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4739.069322] Lustre: 147380:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4741.088404] Lustre: 147380:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788444565/real 1788444565] req@00000000b267de16 x1875319923444096/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788444581 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4741.134420] Lustre: 147380:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4744.174694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4744.200142] Lustre: 147380:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788444568/real 1788444568] req@00000000d123c069 x1875319923444992/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788444584 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4744.439156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4744.459527] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4749.583717] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4750.623922] Lustre: 147380:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788444574/real 1788444574] req@0000000022649d1b x1875319923446016/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788444590 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4750.661808] Lustre: 147380:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4750.811684] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 4751.987425] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4752.014893] Lustre: lustre-OST0000: Recovery over after 0:03, of 3 clients 3 recovered and 0 were evicted. [ 4754.911912] Lustre: 147379:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788444579/real 1788444579] req@000000002adf0769 x1875319923447168/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788444595 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4754.944208] Lustre: 147379:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4755.818742] Lustre: Failing over lustre-MDT0000 [ 4756.224327] Lustre: server umount lustre-MDT0000 complete [ 4760.035201] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4760.050394] Lustre: Skipped 2 previous similar messages [ 4762.119992] LustreError: 149166:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4762.162666] LustreError: 149166:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4765.177845] LustreError: 149171:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4765.196342] LustreError: 149171:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4767.259830] LustreError: 151382:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4770.272185] LustreError: 149171:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4770.294024] LustreError: 149171:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4775.395134] LustreError: 149170:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4775.426945] LustreError: 149170:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4776.415409] Lustre: 147380:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788444600/real 1788444600] req@00000000e073ce69 x1875319923457024/t0(0) o400->MGC192.168.201.127@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788444616 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4776.503926] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4780.838862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4783.562679] LustreError: 149167:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4783.580315] LustreError: 149167:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 4786.413201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4786.444095] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4788.750423] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4790.839676] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 4791.801707] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4791.809222] Lustre: Skipped 1 previous similar message [ 4791.861073] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4791.908677] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4802.019337] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4802.027434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4802.037655] Lustre: Skipped 2 previous similar messages [ 4802.052145] Lustre: Skipped 1 previous similar message [ 4806.661378] Lustre: server umount lustre-OST0000 complete [ 4812.266446] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4812.270327] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4812.282112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4812.308698] Lustre: Skipped 1 previous similar message [ 4817.377555] LustreError: 149171:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4817.386751] LustreError: 149171:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4817.472896] Lustre: server umount lustre-MDT0000 complete [ 4821.717032] LustreError: 149151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444661 with bad export cookie 12026448012862053372 [ 4821.721861] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4821.731096] LustreError: 149151:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4821.980884] Lustre: server umount lustre-MDT0001 complete [ 4836.478203] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 4840.154349] Key type lgssc unregistered [ 4840.474703] LNet: 154780:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4840.478823] LNetError: 154780:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4841.514515] LNet: Removed LNI 192.168.201.127@tcp [ 4842.864450] Key type .llcrypt unregistered [ 4842.873257] Key type ._llcrypt unregistered [ 4888.119180] Key type ._llcrypt registered [ 4888.122418] Key type .llcrypt registered [ 4888.225848] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 4889.822914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4889.843362] alg: No test for adler32 (adler32-zlib) [ 4890.955540] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 4891.362711] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 4893.047177] Key type lgssc registered [ 4894.469957] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4906.756026] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4906.780988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4908.529370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4914.195753] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 4918.756043] LustreError: 156301:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4918.768484] LustreError: 156301:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4923.875253] LustreError: 156302:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4924.773784] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4928.992257] LustreError: 156301:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4929.529624] LustreError: 156330:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4929.544918] LustreError: 156330:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4929.549858] LustreError: 156330:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 4929.902807] Lustre: server umount lustre-MDT0000 complete [ 4947.776266] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 4959.027632] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 5000.124713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5000.400996] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5000.445170] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5000.533926] Lustre: lustre-MDT0000: new disk, initializing [ 5000.612235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5000.632588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5005.190086] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 5016.703107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5016.851131] Lustre: 159638:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5016.939357] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5016.949357] Lustre: Skipped 1 previous similar message [ 5017.067552] Lustre: lustre-MDT0001: new disk, initializing [ 5017.172450] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5017.206510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5017.218543] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5022.651394] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 5036.201891] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5042.009890] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5050.336936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5050.658477] Lustre: lustre-OST0000: new disk, initializing [ 5050.663763] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5050.671828] Lustre: 161607:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5050.765752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5056.598250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5056.618407] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5056.774483] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5057.613486] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 5068.793862] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5073.451514] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5075.506207] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 10:15:14 (1788444914) [ 5353.952322] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5353.970800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5353.982252] Lustre: Skipped 2 previous similar messages [ 5354.003219] Lustre: Skipped 2 previous similar messages [ 5359.072086] LustreError: 159651:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5359.107861] LustreError: 159651:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5360.987057] Lustre: server umount lustre-MDT0000 complete [ 5364.191901] LustreError: 162489:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5364.228705] LustreError: 162489:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5365.642133] LustreError: 159631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445205 with bad export cookie 17573953260192575775 [ 5365.642786] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5365.656371] LustreError: 159631:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5369.326989] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5369.348072] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5372.226090] Lustre: server umount lustre-MDT0001 complete [ 5382.635699] Lustre: server umount lustre-OST0000 complete [ 5394.970370] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 5409.129897] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 5452.395937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5452.970877] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5453.054786] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5453.382274] Lustre: lustre-MDT0000: new disk, initializing [ 5453.576467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5453.624334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5459.910686] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 5474.692149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5474.962081] Lustre: 166287:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5475.059263] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5475.063315] Lustre: Skipped 1 previous similar message [ 5475.174200] Lustre: lustre-MDT0001: new disk, initializing [ 5475.390747] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5475.455410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5475.476191] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5480.977291] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 5491.526281] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5498.661730] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5506.204256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5506.502570] Lustre: lustre-OST0000: new disk, initializing [ 5506.506273] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5506.512741] Lustre: 168257:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5506.603380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5513.290723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5513.298484] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5513.478764] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5513.643155] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 5525.734566] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5531.806327] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5532.025844] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5537.363265] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5537.699878] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5544.434382] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5544.446687] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5544.493560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5544.495902] Lustre: Skipped 1 previous similar message [ 5549.544663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5549.564083] Lustre: Skipped 1 previous similar message [ 5554.143224] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5554.453777] Lustre: server umount lustre-OST0000 complete [ 5562.337419] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5562.352761] LustreError: Skipped 1 previous similar message [ 5562.364972] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5562.404742] Lustre: Skipped 1 previous similar message [ 5562.426106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5568.032450] Lustre: server umount lustre-MDT0000 complete [ 5570.016509] LustreError: 166301:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5570.046447] LustreError: 166301:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 5573.216700] LustreError: 166282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445413 with bad export cookie 17573953260192734605 [ 5573.221177] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5573.225716] LustreError: 166282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5573.887655] Lustre: server umount lustre-MDT0001 complete [ 5584.223978] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:23:43 (1788445423) [ 5603.124296] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 5614.813188] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 5654.996886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5655.217376] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5655.255362] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5655.353690] Lustre: lustre-MDT0000: new disk, initializing [ 5655.450060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5655.462491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5660.112610] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 5671.085767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5671.209825] Lustre: 173018:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5671.445637] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5676.881923] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 5687.416615] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5692.857494] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5701.586940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5702.008648] Lustre: lustre-OST0000: new disk, initializing [ 5702.011607] Lustre: Skipped 1 previous similar message [ 5702.015456] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5702.021262] Lustre: Skipped 2 previous similar messages [ 5702.026544] Lustre: 174995:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5708.425435] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 5709.345430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5709.357393] Lustre: Skipped 1 previous similar message [ 5709.367621] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5709.496861] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5718.147657] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5730.275141] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5730.279088] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5730.329468] Lustre: Skipped 1 previous similar message [ 5730.331649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5730.333613] Lustre: Skipped 2 previous similar messages [ 5730.432113] LustreError: Skipped 1 previous similar message [ 5731.774355] Lustre: server umount lustre-OST0000 complete [ 5737.955656] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5740.521943] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5740.570123] Lustre: Skipped 2 previous similar messages [ 5743.810225] Lustre: server umount lustre-MDT0000 complete [ 5745.649583] LustreError: 174187:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5750.440516] LustreError: 173010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445590 with bad export cookie 17573953260192735487 [ 5750.448964] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5750.452606] LustreError: 173010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5751.359433] Lustre: server umount lustre-MDT0001 complete [ 5764.906400] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:26:44 (1788445604) [ 5782.939497] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 5793.748184] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 5832.434670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5832.675251] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5832.706733] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5832.792041] Lustre: lustre-MDT0000: new disk, initializing [ 5832.889191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5832.896079] Lustre: Skipped 2 previous similar messages [ 5832.915919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5837.686178] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 5849.266265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5849.521211] Lustre: 179335:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5849.778991] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5854.616926] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 5865.738494] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5873.983215] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5882.142036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5882.546569] Lustre: 181314:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5888.597625] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5888.780595] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5890.801582] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 5901.848316] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5914.594124] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5914.606541] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5914.611885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5914.620960] Lustre: Skipped 1 previous similar message [ 5914.635666] Lustre: Skipped 5 previous similar messages [ 5916.254492] Lustre: server umount lustre-OST0000 complete [ 5926.967168] Lustre: server umount lustre-MDT0000 complete [ 5929.976742] LustreError: 179343:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5930.010277] LustreError: 179343:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5931.269080] LustreError: 179329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445771 with bad export cookie 17573953260192736572 [ 5931.272732] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5931.282364] LustreError: 179329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5931.813273] Lustre: server umount lustre-MDT0001 complete [ 5943.432440] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 5946.895996] Key type lgssc unregistered [ 5947.233954] LNet: 183171:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5947.240752] LNetError: 183171:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5947.258916] LNet: Removed LNI 192.168.201.127@tcp [ 5948.169267] Key type .llcrypt unregistered [ 5948.171774] Key type ._llcrypt unregistered [ 5973.151388] Key type ._llcrypt registered [ 5973.153740] Key type .llcrypt registered [ 5973.249394] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 5974.806177] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5974.869783] alg: No test for adler32 (adler32-zlib) [ 5975.977861] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 5976.340772] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 5978.087863] Key type lgssc registered [ 5979.475887] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5988.019149] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5988.049449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5989.889895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5994.783893] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 6000.098096] LustreError: 184586:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6000.130938] LustreError: 184586:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6005.072926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6005.223772] LustreError: 184585:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6005.563294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6010.351201] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 6018.453159] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6025.039859] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6033.556333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6034.349820] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6035.388674] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6041.719768] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 6050.719929] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6073.857195] Lustre: server umount lustre-OST0000 complete [ 6081.509972] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6081.520201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6081.533772] Lustre: Skipped 1 previous similar message [ 6086.631942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6086.638118] Lustre: Skipped 2 previous similar messages [ 6088.735718] Lustre: server umount lustre-MDT0000 complete [ 6091.746732] LustreError: 184581:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6091.787140] LustreError: 184581:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6092.452198] Lustre: 183893:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788445916/real 1788445916] req@00000000bf748e97 x1875321359990656/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788445932 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6092.480565] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6093.001276] LustreError: 184564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445933 with bad export cookie 6461242396565774503 [ 6093.020796] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6093.025829] LustreError: 184564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6093.632060] Lustre: server umount lustre-MDT0001 complete [ 6105.627287] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 6108.525859] Key type lgssc unregistered [ 6108.915142] LNet: 188143:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6108.921129] LNetError: 188143:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6109.999699] LNet: Removed LNI 192.168.201.127@tcp [ 6111.104608] Key type .llcrypt unregistered [ 6111.109526] Key type ._llcrypt unregistered [ 6137.542133] Key type ._llcrypt registered [ 6137.545993] Key type .llcrypt registered [ 6137.685694] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 6138.975109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6139.134849] alg: No test for adler32 (adler32-zlib) [ 6140.289074] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 6140.676164] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 6142.439220] Key type lgssc registered [ 6143.883487] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6151.545574] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6151.557107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6153.057678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6159.908062] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 6163.427353] LustreError: 189555:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6163.455391] LustreError: 189555:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6168.547468] LustreError: 189556:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6172.077770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6172.584647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6178.558671] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 6187.374888] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6194.761091] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6203.269893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6203.745406] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6210.609508] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 6221.581684] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6223.242713] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6233.571150] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6233.626497] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6233.631133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6238.265564] Lustre: lustre-OST0000: Not available for connect from 192.168.201.27@tcp (stopping) [ 6238.271992] Lustre: Skipped 1 previous similar message [ 6240.007866] Lustre: server umount lustre-OST0000 complete [ 6249.881812] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff953d44c64800.ost_server_uuid 50 [ 6252.411282] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6268.385942] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6268.402369] Lustre: Skipped 1 previous similar message [ 6268.410309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6268.416429] Lustre: Skipped 3 previous similar messages [ 6273.046462] Lustre: server umount lustre-MDT0000 complete [ 6273.516434] LustreError: 190284:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6273.526061] LustreError: 190284:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6276.591425] LustreError: 189536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446116 with bad export cookie 17061426077822660301 [ 6276.604797] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6277.113501] Lustre: server umount lustre-MDT0001 complete [ 6285.630985] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 10:35:25 (1788446125) [ 6292.033765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6292.424982] LustreError: 193474:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6292.486171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6296.444704] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 6297.571778] LustreError: 193475:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6304.620937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6304.942092] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6309.621250] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 6317.876605] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6325.600610] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6333.474962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6334.109135] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6337.201728] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6340.871569] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 6350.539029] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6359.837190] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6360.068951] Lustre: lustre-OST0001: new disk, initializing [ 6360.075513] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6360.093923] Lustre: 196319:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6360.271403] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6368.569178] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 6368.969873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6368.996818] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6369.127627] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6378.232602] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6381.550363] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6381.560852] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6381.579187] Lustre: Skipped 1 previous similar message [ 6381.588518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6383.584765] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6383.592954] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6387.421791] Lustre: server umount lustre-OST0000 complete [ 6389.257777] LustreError: 195299:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6389.276986] LustreError: 195299:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6393.826832] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6394.002692] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6397.824384] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6397.988446] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6410.222108] LustreError: 195298:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6410.249855] LustreError: 195298:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 6415.329249] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6415.340811] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6415.369673] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6415.383464] Lustre: Skipped 5 previous similar messages [ 6418.316621] Lustre: server umount lustre-OST0001 complete [ 6423.009438] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6425.569182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6425.573645] Lustre: Skipped 1 previous similar message [ 6425.584395] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6425.616456] Lustre: Skipped 2 previous similar messages [ 6437.344789] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6437.519729] Lustre: server umount lustre-MDT0000 complete [ 6441.439572] LustreError: 193456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446281 with bad export cookie 17061426077822661372 [ 6441.441789] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6441.448103] LustreError: 193456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6441.734800] Lustre: server umount lustre-MDT0001 complete [ 6464.722052] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 10:38:23 (1788446303) [ 6471.484025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6471.688651] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6471.720974] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6471.910185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6476.306724] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 6485.537087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6485.677934] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6485.735857] Lustre: 199764:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6490.902598] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 6499.144126] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6505.454042] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6512.755806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6513.046523] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6519.721043] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 6520.470286] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6528.090292] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6535.668534] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6535.750977] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6536.054719] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6536.059676] Lustre: Skipped 2 previous similar messages [ 6542.462515] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 6544.580200] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6549.507319] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6555.115839] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6555.144841] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6555.156265] Lustre: Skipped 6 previous similar messages [ 6558.213295] Lustre: server umount lustre-OST0000 complete [ 6560.233413] LustreError: 201615:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6560.246545] LustreError: 201615:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6575.076184] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6581.441353] Lustre: server umount lustre-OST0001 complete [ 6590.946048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6590.957926] Lustre: Skipped 8 previous similar messages [ 6600.159209] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6600.440354] Lustre: server umount lustre-MDT0000 complete [ 6604.132978] LustreError: 200521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446444 with bad export cookie 17061426077822662779 [ 6604.134204] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6604.139578] LustreError: 200521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6604.429874] Lustre: server umount lustre-MDT0001 complete [ 6628.611685] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 10:41:08 (1788446468) [ 6643.037158] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 6652.628504] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 6691.538120] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6691.761963] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6691.791138] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6691.870931] Lustre: lustre-MDT0000: new disk, initializing [ 6691.939151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6691.950833] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6696.475234] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 6707.089161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6707.233831] Lustre: 207859:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6707.285509] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6707.290548] Lustre: Skipped 1 previous similar message [ 6707.367472] Lustre: lustre-MDT0001: new disk, initializing [ 6707.485470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6707.497408] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6711.491304] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 6720.905390] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6727.156556] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6733.819122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6734.077306] Lustre: lustre-OST0000: new disk, initializing [ 6734.080538] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6734.085475] Lustre: 209832:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6740.451632] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 6744.253804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6744.272265] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6744.459542] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6751.775618] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6757.890601] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6758.126554] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6763.464590] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6763.717681] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6770.157080] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6770.170029] LustreError: Skipped 1 previous similar message [ 6770.178986] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6770.195529] Lustre: Skipped 5 previous similar messages [ 6770.200966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6770.213606] Lustre: Skipped 3 previous similar messages [ 6771.844019] Lustre: server umount lustre-OST0000 complete [ 6781.712421] Lustre: server umount lustre-MDT0000 complete [ 6784.947903] LustreError: 209831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446625 with bad export cookie 17061426077822664564 [ 6784.949072] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6784.961369] LustreError: 209831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6785.226304] Lustre: server umount lustre-MDT0001 complete [ 6793.192618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6793.835655] LustreError: 212049:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6793.854048] LustreError: 212049:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 6797.586543] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 6806.973410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6811.273204] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 6820.264301] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6828.094412] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6836.312824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6843.401279] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 6852.182758] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6857.528457] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6857.735168] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6862.573190] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6862.811323] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6866.401208] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6866.425080] LustreError: Skipped 2 previous similar messages [ 6866.433083] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6866.445280] Lustre: Skipped 3 previous similar messages [ 6879.199483] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6879.353373] Lustre: server umount lustre-OST0000 complete [ 6885.286595] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6890.039170] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6918.820625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6925.512733] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 6932.356896] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6932.458586] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6932.469910] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6932.472670] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6932.487565] Lustre: Skipped 1 previous similar message [ 6937.193499] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 6942.693250] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6942.700419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6942.701192] LustreError: Skipped 2 previous similar messages [ 6942.708173] Lustre: Skipped 10 previous similar messages [ 6947.391934] Lustre: server umount lustre-OST0000 complete [ 6957.502996] Lustre: server umount lustre-MDT0000 complete [ 6958.049792] LustreError: 212044:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6958.078344] LustreError: 212044:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6961.100835] LustreError: 217144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446801 with bad export cookie 17061426077822665446 [ 6961.105996] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6961.110948] LustreError: 217144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6961.478699] Lustre: server umount lustre-MDT0001 complete [ 6969.753946] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 10:46:49 (1788446809) [ 6977.325497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6977.974499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6977.983862] Lustre: Skipped 6 previous similar messages [ 6982.838206] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 6991.835705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6996.969568] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7004.738169] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7011.243446] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7017.526057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7023.901841] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7030.838418] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7035.343236] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7035.582931] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7039.635288] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7039.805555] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7046.142817] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7046.336731] Lustre: lustre-OST0001: new disk, initializing [ 7046.339355] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7046.342782] Lustre: 221388:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7052.063186] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7055.992554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7056.004832] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7056.135901] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7061.687645] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7066.832796] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7067.001861] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7070.940928] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7071.180487] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7076.835690] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7076.839647] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7076.861031] LustreError: Skipped 1 previous similar message [ 7076.877657] Lustre: Skipped 6 previous similar messages [ 7079.280417] Lustre: server umount lustre-OST0001 complete [ 7085.422086] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7085.618471] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7090.701433] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7090.987707] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7121.479526] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7127.012789] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7127.020269] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7127.024864] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7127.033469] Lustre: Skipped 1 previous similar message [ 7127.205441] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7133.148341] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7134.509260] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7142.185111] Lustre: server umount lustre-OST0001 complete [ 7165.235745] Lustre: server umount lustre-OST0000 complete [ 7183.327135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7183.549260] Lustre: server umount lustre-MDT0000 complete [ 7186.672696] LustreError: 217994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447026 with bad export cookie 17061426077822666412 [ 7186.695434] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7186.696837] LustreError: 217994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7193.055355] Lustre: server umount lustre-MDT0001 complete [ 7214.634221] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 10:50:53 (1788447053) [ 7221.277283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7221.462893] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7221.502974] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7226.905615] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7235.320878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7235.489543] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7235.561809] Lustre: 226630:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7240.277845] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7249.148899] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7256.711815] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7264.860926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7265.034708] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7271.424888] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7279.875171] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7289.495966] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7289.580382] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7296.266538] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7305.157785] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7306.844671] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7312.916120] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7313.249369] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7318.817807] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7319.190606] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7326.623924] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff953d46e84000.ost_server_uuid 50 [ 7328.140273] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff953d46e84000.ost_server_uuid in IDLE state after 0 sec [ 7329.148703] Lustre: Permanently deactivating lustre-OST0001 [ 7329.166044] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7332.386359] Lustre: Permanently reactivating lustre-OST0001 [ 7337.962392] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7337.963475] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7337.982194] Lustre: Skipped 6 previous similar messages [ 7337.984731] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7337.984740] Lustre: Skipped 18 previous similar messages [ 7338.002919] LustreError: Skipped 6 previous similar messages [ 7343.872488] Lustre: server umount lustre-OST0001 complete [ 7348.200204] LustreError: 231196:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7348.221056] LustreError: 231196:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 37 previous similar messages [ 7353.846645] Lustre: server umount lustre-OST0000 complete [ 7363.925408] Lustre: server umount lustre-MDT0000 complete [ 7367.333925] LustreError: 226624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447207 with bad export cookie 17061426077822667693 [ 7367.340984] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7367.349996] LustreError: 226624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7367.894745] Lustre: server umount lustre-MDT0001 complete [ 7392.436661] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 10:53:51 (1788447231) [ 7400.831230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7401.077786] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7401.139118] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7401.155798] Lustre: Skipped 5 previous similar messages [ 7405.493933] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7414.597363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7414.948858] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7414.973591] Lustre: 233327:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7420.210127] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7429.984585] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7437.670936] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7445.019698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7445.254112] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7445.300876] Lustre: Permanently deactivating lustre-OST0000 [ 7445.333862] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7452.055836] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7454.799234] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7460.268279] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7467.080928] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7467.155992] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7473.145580] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7473.320839] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7481.080707] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7482.718823] Lustre: Permanently reactivating lustre-OST0000 [ 7482.735377] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7482.746126] Lustre: Skipped 2 previous similar messages [ 7492.772093] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7492.786147] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 37 seconds. I think it's dead, and I am evicting it. exp 00000000ac601d58, cur 1788447332 deadline 1788447325 last 1788447295 [ 7492.791953] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7492.813545] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7492.917441] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7504.777117] Lustre: server umount lustre-OST0001 complete [ 7523.295184] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7523.445573] Lustre: server umount lustre-OST0000 complete [ 7532.940325] Lustre: server umount lustre-MDT0000 complete [ 7536.345816] LustreError: 236922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447376 with bad export cookie 17061426077822669345 [ 7536.347112] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7536.354955] LustreError: 236922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7536.614731] Lustre: server umount lustre-MDT0001 complete [ 7548.494418] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 7550.828401] Key type lgssc unregistered [ 7551.030269] LNet: 238205:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7551.040398] LNetError: 238205:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7552.105674] LNet: Removed LNI 192.168.201.127@tcp [ 7552.999119] Key type .llcrypt unregistered [ 7553.001561] Key type ._llcrypt unregistered [ 7577.997321] Key type ._llcrypt registered [ 7577.999657] Key type .llcrypt registered [ 7578.149780] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 7579.129375] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7579.277765] alg: No test for adler32 (adler32-zlib) [ 7580.389799] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 7580.589463] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 7582.239300] Key type lgssc registered [ 7583.048768] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7591.839388] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7591.853863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7593.276713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7598.544456] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7603.686251] LustreError: 239669:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7603.706727] LustreError: 239669:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7607.775621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7607.897137] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7607.918722] Lustre: 239658:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7607.926634] Lustre: Permanently deactivating lustre-MDT0001 [ 7607.936396] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7608.149509] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7612.962180] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7618.123493] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7622.881860] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7629.566592] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7637.465347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7637.904292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7638.901672] LustreError: 241511:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7638.928974] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7644.130934] LustreError: 241511:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7644.517912] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7649.278724] LustreError: 241509:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7652.109692] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7653.348781] LustreError: 241509:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7653.377783] LustreError: 241509:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7659.217980] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7659.481117] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7664.616812] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7664.627277] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7664.849261] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7672.078693] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7674.988558] Lustre: Permanently reactivating lustre-MDT0001 [ 7675.010451] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7675.012989] Lustre: Skipped 2 previous similar messages [ 7678.584125] Lustre: lustre-MDT0001: Client fd748112-bee9-4a04-8945-441231f26c55 (at 192.168.201.27@tcp) reconnecting [ 7684.332144] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7684.347926] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7684.373847] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7684.389180] LustreError: 243383:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7684.395362] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7684.400071] LustreError: 243383:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7684.419729] LustreError: 243383:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.127@tcp: cfg command failed: rc = -108 [ 7684.433852] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7684.433852] [ 7684.441026] LustreError: 239660:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7687.164548] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7689.099534] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7695.457248] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7695.657440] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7700.623667] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7700.869874] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7703.032237] Lustre: Permanently deactivating lustre-MDT0001 [ 7703.045800] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7703.054741] Lustre: Skipped 2 previous similar messages [ 7709.303215] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7719.436995] Lustre: server umount lustre-MDT0000 complete [ 7720.950712] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7720.952740] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7720.961971] LustreError: 240414:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7720.961986] LustreError: 240414:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7721.016791] Lustre: Skipped 2 previous similar messages [ 7723.833877] LustreError: 243217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447564 with bad export cookie 14481061541251284609 [ 7723.837256] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7723.855201] LustreError: 243217:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7724.260847] Lustre: server umount lustre-MDT0001 complete [ 7728.160627] Lustre: server umount lustre-OST0000 complete [ 7740.677477] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 10:59:39 (1788447579) [ 7753.098887] Lustre: server umount lustre-OST0001 complete [ 7759.502069] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 7769.396524] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 7809.535876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7809.804912] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7809.815480] Lustre: Skipped 2 previous similar messages [ 7809.846157] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7809.954311] Lustre: lustre-MDT0000: new disk, initializing [ 7810.096478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7810.117434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7814.979553] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7826.152654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7826.290330] Lustre: 247952:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7826.417891] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7826.422522] Lustre: Skipped 1 previous similar message [ 7826.557183] Lustre: lustre-MDT0001: new disk, initializing [ 7826.740205] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7826.767057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7826.781701] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7830.896155] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7841.045658] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7847.142300] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7853.820203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7854.230796] Lustre: lustre-OST0000: new disk, initializing [ 7854.234644] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7854.239701] Lustre: 249931:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7854.314650] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7860.219597] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7862.375421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7862.384601] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7862.493435] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7868.869217] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7871.170180] LustreError: 249939:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7879.764634] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7879.971242] Lustre: lustre-OST0001: new disk, initializing [ 7879.974673] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7879.980301] Lustre: 251044:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7881.176049] LustreError: 249939:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7881.214228] LustreError: 250602:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7885.370759] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 7886.372468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7886.384332] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7886.446064] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7891.303379] LustreError: 250602:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7895.458331] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7897.102321] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7901.222821] Lustre: lustre-OST0001: Not available for connect from 192.168.201.27@tcp (stopping) [ 7901.670697] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7901.675647] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7901.696762] Lustre: Skipped 1 previous similar message [ 7905.447388] Lustre: server umount lustre-OST0001 complete [ 7906.317063] LustreError: 249916:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7906.333564] LustreError: 249916:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7922.157026] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7922.175888] Lustre: Skipped 1 previous similar message [ 7922.190830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7922.193476] Lustre: Skipped 2 previous similar messages [ 7927.271881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7927.279294] Lustre: Skipped 1 previous similar message [ 7927.608350] Lustre: server umount lustre-OST0000 complete [ 7932.384553] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7932.406330] Lustre: Skipped 1 previous similar message [ 7932.411896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7932.418501] Lustre: Skipped 1 previous similar message [ 7937.528045] Lustre: server umount lustre-MDT0000 complete [ 7941.272656] LustreError: 247943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447781 with bad export cookie 14481061541251287206 [ 7941.279357] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7941.281814] LustreError: 247943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7947.701830] Lustre: server umount lustre-MDT0001 complete [ 7959.574475] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 7962.711773] Key type lgssc unregistered [ 7962.889829] LNet: 253078:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7962.895542] LNetError: 253078:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7962.906684] LNet: Removed LNI 192.168.201.127@tcp [ 7963.710315] Key type .llcrypt unregistered [ 7963.712088] Key type ._llcrypt unregistered [ 8007.258908] Key type ._llcrypt registered [ 8007.270460] Key type .llcrypt registered [ 8007.365699] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 8008.280122] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8008.362348] alg: No test for adler32 (adler32-zlib) [ 8009.569867] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8009.762056] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 8011.391165] Key type lgssc registered [ 8012.462827] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8019.607303] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8019.620832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8020.862732] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8020.882642] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8021.060206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8025.587978] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 8033.906410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8033.983445] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8034.000538] Lustre: 255284:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8034.274566] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8038.505345] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 8046.255434] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8051.814481] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8058.514183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8058.779661] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8059.099446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8065.172413] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8066.587378] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 8074.195370] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8131.556207] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8131.562063] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8131.570866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8132.064096] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8132.077171] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8132.085790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8132.088987] Lustre: Skipped 1 previous similar message [ 8134.089704] Lustre: server umount lustre-OST0000 complete [ 8138.579077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8150.329221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8150.533765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8152.619321] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8152.650659] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8156.888479] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 8157.732396] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8157.757187] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8165.446896] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8208.875155] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8208.875241] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8208.878459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8208.927893] Lustre: Skipped 1 previous similar message [ 8212.204763] Lustre: server umount lustre-OST0000 complete [ 8218.594535] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8218.605565] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8218.617773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8218.622847] Lustre: Skipped 1 previous similar message [ 8224.227185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8224.238861] Lustre: Skipped 3 previous similar messages [ 8230.879211] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8231.117922] Lustre: server umount lustre-MDT0000 complete [ 8235.210470] LustreError: 255278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448075 with bad export cookie 18399448458165844752 [ 8235.239801] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8235.249046] LustreError: 255278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8235.792528] Lustre: server umount lustre-MDT0001 complete [ 8246.297393] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 8249.300340] Key type lgssc unregistered [ 8249.590917] LNet: 260706:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8249.596453] LNetError: 260706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8249.607892] LNet: Removed LNI 192.168.201.127@tcp [ 8250.489916] Key type .llcrypt unregistered [ 8250.501403] Key type ._llcrypt unregistered [ 8275.756819] Key type ._llcrypt registered [ 8275.759426] Key type .llcrypt registered [ 8275.865482] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 8277.159164] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8277.269095] alg: No test for adler32 (adler32-zlib) [ 8278.338915] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8278.553789] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 8280.191179] Key type lgssc registered [ 8281.252228] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8287.172329] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8287.190964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8288.501731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8292.332902] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 8297.954141] LustreError: 262120:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8297.970594] LustreError: 262120:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8300.045287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8300.377892] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8304.708977] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 8312.001061] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8318.380308] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8325.884518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8326.514343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8333.605798] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 8334.726563] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8341.376152] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8370.656648] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8370.659898] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8370.677891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8370.682778] Lustre: Skipped 1 previous similar message [ 8370.710521] Lustre: Skipped 1 previous similar message [ 8372.723975] Lustre: server umount lustre-OST0000 complete [ 8377.312520] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8377.325623] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8377.340142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8380.900140] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8380.907045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8380.925444] Lustre: Skipped 1 previous similar message [ 8382.602818] Lustre: server umount lustre-MDT0000 complete [ 8386.018985] LustreError: 262847:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8386.041848] LustreError: 262847:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8386.162294] LustreError: 262099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448226 with bad export cookie 2416745339784576332 [ 8386.168087] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8386.169604] LustreError: 262099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8386.375489] Lustre: server umount lustre-MDT0001 complete [ 8395.871483] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 8398.420753] Key type lgssc unregistered [ 8398.700110] LNet: 266429:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8398.712236] LNetError: 266429:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8398.738547] LNet: Removed LNI 192.168.201.127@tcp [ 8399.572123] Key type .llcrypt unregistered [ 8399.575515] Key type ._llcrypt unregistered [ 8418.893535] Key type ._llcrypt registered [ 8418.895916] Key type .llcrypt registered [ 8418.981099] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 8419.958424] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8420.241182] alg: No test for adler32 (adler32-zlib) [ 8421.371422] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8421.674768] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 8423.352294] Key type lgssc registered [ 8424.421984] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8432.432155] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8432.476087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8434.060108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8438.450987] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 8444.399133] LustreError: 267647:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8444.413097] LustreError: 267647:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8447.802772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8448.302139] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8453.865747] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 8461.987370] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8468.037133] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8475.384512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8475.643817] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8479.732605] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8481.002099] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 8489.326889] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8503.265931] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8503.268081] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8503.274827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8503.282871] Lustre: Skipped 1 previous similar message [ 8504.699411] Lustre: server umount lustre-OST0000 complete [ 8508.384528] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8508.405464] Lustre: Skipped 1 previous similar message [ 8508.415705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8513.504625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8513.516896] Lustre: Skipped 2 previous similar messages [ 8514.518990] Lustre: server umount lustre-MDT0000 complete [ 8518.627070] LustreError: 267647:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8518.657482] LustreError: 267647:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8518.932411] LustreError: 267627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448359 with bad export cookie 5334468818663180617 [ 8518.943596] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8518.965883] LustreError: 267627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8519.816774] Lustre: server umount lustre-MDT0001 complete [ 8534.089903] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 8537.950928] Key type lgssc unregistered [ 8538.616223] LNet: 271323:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8538.622292] LNetError: 271323:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8538.643229] LNet: Removed LNI 192.168.201.127@tcp [ 8540.166282] Key type .llcrypt unregistered [ 8540.167961] Key type ._llcrypt unregistered [ 8570.063680] Key type ._llcrypt registered [ 8570.066427] Key type .llcrypt registered [ 8570.210576] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 8571.884933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8571.981567] alg: No test for adler32 (adler32-zlib) [ 8573.543538] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8573.999536] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 8575.815170] Key type lgssc registered [ 8577.422162] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8585.902361] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8585.928583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8587.279841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8592.368330] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 8597.474736] LustreError: 272740:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8597.495171] LustreError: 272740:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8601.890381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8602.235140] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8605.898430] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 8614.123590] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8620.372893] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8627.953802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8628.202108] LustreError: 274560:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8628.351988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8633.529463] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8633.834577] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 8642.320614] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8669.666113] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8669.686657] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8669.689197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8669.699477] Lustre: Skipped 1 previous similar message [ 8671.975946] Lustre: server umount lustre-OST0000 complete [ 8678.885367] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8678.889749] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8678.898822] Lustre: Skipped 1 previous similar message [ 8678.904126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8679.905372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8679.907701] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8682.188590] Lustre: server umount lustre-MDT0000 complete [ 8685.024341] LustreError: 272737:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8685.046775] LustreError: 272737:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8685.906732] LustreError: 272722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448526 with bad export cookie 16343402805000701818 [ 8685.917614] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8685.920049] LustreError: 272722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8686.194135] Lustre: server umount lustre-MDT0001 complete [ 8697.481963] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 8700.124460] Key type lgssc unregistered [ 8700.417773] LNet: 277043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8700.424640] LNetError: 277043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8700.454212] LNet: Removed LNI 192.168.201.127@tcp [ 8701.173112] Key type .llcrypt unregistered [ 8701.175313] Key type ._llcrypt unregistered [ 8718.087450] Key type ._llcrypt registered [ 8718.091572] Key type .llcrypt registered [ 8718.203211] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 8719.127748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8719.294417] alg: No test for adler32 (adler32-zlib) [ 8720.348913] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8720.615526] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 8722.279191] Key type lgssc registered [ 8723.226973] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8730.361230] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8730.399123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8731.766885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8735.651134] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 8741.354324] LustreError: 278274:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8741.377994] LustreError: 278274:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8744.184230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8744.567230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8748.795414] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 8756.329256] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8761.490137] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8767.167050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8767.479686] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8773.290602] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 8776.700225] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8780.546340] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8795.105554] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8795.115439] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8795.124084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8797.161207] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8797.163683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8797.182351] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8797.217584] Lustre: Skipped 1 previous similar message [ 8798.524705] Lustre: server umount lustre-OST0000 complete [ 8802.274716] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8802.276514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8802.294533] Lustre: Skipped 1 previous similar message [ 8802.326934] Lustre: Skipped 1 previous similar message [ 8807.396099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8807.412935] Lustre: Skipped 1 previous similar message [ 8808.339352] Lustre: server umount lustre-MDT0000 complete [ 8812.515282] LustreError: 278272:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8812.536206] LustreError: 278272:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8812.660030] LustreError: 278240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448652 with bad export cookie 15961566137757693579 [ 8812.670372] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8812.676933] LustreError: 278240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8813.140769] Lustre: server umount lustre-MDT0001 complete [ 8824.508294] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 8827.124661] Key type lgssc unregistered [ 8827.367654] LNet: 281973:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8827.372110] LNetError: 281973:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8827.381955] LNet: Removed LNI 192.168.201.127@tcp [ 8828.169862] Key type .llcrypt unregistered [ 8828.173312] Key type ._llcrypt unregistered [ 8841.220307] Key type ._llcrypt registered [ 8841.222278] Key type .llcrypt registered [ 8841.308389] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8858.680344] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 8874.745657] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 8875.767507] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8875.876344] alg: No test for adler32 (adler32-zlib) [ 8876.938819] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8877.174902] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 8878.871173] Key type lgssc registered [ 8880.164465] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8914.395337] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8914.421303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8915.648189] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8915.670610] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8915.716775] Lustre: lustre-MDT0000: new disk, initializing [ 8915.778709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8915.810057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8920.063706] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 8930.570126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8930.750533] Lustre: 285524:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8930.848690] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8930.857083] Lustre: Skipped 1 previous similar message [ 8930.964288] Lustre: lustre-MDT0001: new disk, initializing [ 8931.044623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8931.081057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8931.093361] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8935.760578] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 8944.732747] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8950.707706] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8957.815914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8958.230644] Lustre: lustre-OST0000: new disk, initializing [ 8958.238493] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8958.247720] Lustre: 287494:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8958.340972] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8963.699225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8963.718889] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8963.906749] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8965.949490] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 8975.850614] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8980.474694] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8980.775603] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8985.411596] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8985.625084] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8989.667209] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8989.667373] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8989.686197] LustreError: Skipped 1 previous similar message [ 8989.689551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8989.716927] Lustre: Skipped 1 previous similar message [ 8989.745072] Lustre: Skipped 1 previous similar message [ 8993.516640] Lustre: server umount lustre-OST0000 complete [ 8997.857286] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8997.868909] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8997.886182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8999.904577] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8999.912188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8999.929708] Lustre: Skipped 1 previous similar message [ 9003.888468] Lustre: server umount lustre-MDT0000 complete [ 9007.408882] LustreError: 285517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448847 with bad export cookie 3893021360251605627 [ 9007.416480] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9007.428647] LustreError: 285517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9007.784709] Lustre: server umount lustre-MDT0001 complete [ 9016.267850] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 11:20:55 (1788448855) [ 9023.096896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9023.472843] LustreError: 289906:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9023.483078] LustreError: 289906:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9023.532399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9027.576993] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 9028.577674] LustreError: 289907:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9033.703679] LustreError: 289906:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9035.676827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9036.113403] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9041.133197] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 9049.045405] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9055.111378] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9062.733909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9063.026571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9069.979668] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 9078.411883] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9094.125676] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9094.140472] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9094.158451] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9097.186678] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9098.421262] Lustre: server umount lustre-OST0000 complete [ 9102.818202] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9102.829738] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9102.841176] Lustre: Skipped 1 previous similar message [ 9102.856287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9102.864746] Lustre: Skipped 2 previous similar messages [ 9108.818829] Lustre: server umount lustre-MDT0000 complete [ 9112.146682] LustreError: 289886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448952 with bad export cookie 3893021360251606516 [ 9112.157223] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9112.161081] LustreError: 289886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9112.599379] Lustre: server umount lustre-MDT0001 complete [ 9122.896391] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 9125.218449] Key type lgssc unregistered [ 9125.519703] LNet: 293463:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9125.531579] LNetError: 293463:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9126.566646] LNet: Removed LNI 192.168.201.127@tcp [ 9127.509415] Key type .llcrypt unregistered [ 9127.512634] Key type ._llcrypt unregistered [ 9155.746945] Key type ._llcrypt registered [ 9155.752511] Key type .llcrypt registered [ 9155.877243] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 9170.700826] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 9172.133604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9172.182817] alg: No test for adler32 (adler32-zlib) [ 9173.181774] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 9173.372763] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 9175.023238] Key type lgssc registered [ 9176.651746] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9227.369857] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9227.396445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9228.553237] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9228.571123] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9228.617191] Lustre: lustre-MDT0000: new disk, initializing [ 9228.662545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9228.677182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9232.371357] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 9242.868424] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9242.951461] Lustre: 297285:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9242.984900] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9242.988086] Lustre: Skipped 1 previous similar message [ 9243.038993] Lustre: lustre-MDT0001: new disk, initializing [ 9243.121580] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9243.156906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9243.167210] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9248.199296] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 9260.166782] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9266.944414] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9274.616268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9275.062994] Lustre: lustre-OST2710: new disk, initializing [ 9275.077108] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9275.092948] Lustre: 299255:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9275.227356] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9280.649515] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 9281.623830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9281.629617] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9281.830368] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9291.336827] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9298.663640] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9298.869098] Lustre: lustre-OST03e8: new disk, initializing [ 9298.875903] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9298.890841] Lustre: 300320:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9298.984557] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9305.787176] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 9308.789939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9308.804510] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9308.941837] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9313.686700] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9322.713857] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9322.994410] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9327.959822] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9328.171257] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9332.882691] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9333.035681] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9337.445962] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9337.635331] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9344.998396] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9345.007523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9345.022102] Lustre: Skipped 3 previous similar messages [ 9345.032603] Lustre: Skipped 3 previous similar messages [ 9346.985591] Lustre: server umount lustre-MDT0000 complete [ 9350.226444] LustreError: 299254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449190 with bad export cookie 10449832667480431684 [ 9350.228731] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9350.235783] LustreError: 299254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9350.486505] Lustre: server umount lustre-MDT0001 complete [ 9364.642073] Lustre: server umount lustre-OST2710 complete [ 9378.330116] Lustre: server umount lustre-OST03e8 complete [ 9401.554236] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [ 9415.558883] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 9456.073451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9456.380326] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9456.405626] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9456.493708] Lustre: lustre-MDT0000: new disk, initializing [ 9456.612905] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9456.634214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9461.086624] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 9471.401682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9471.540563] Lustre: 305570:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9471.575509] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9471.578524] Lustre: Skipped 1 previous similar message [ 9471.630924] Lustre: lustre-MDT0001: new disk, initializing [ 9471.691435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9471.727325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9471.740561] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9475.820851] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 9485.181785] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9491.292477] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9498.208185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9498.454393] Lustre: lustre-OST0000: new disk, initializing [ 9498.458587] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9498.466793] Lustre: 307538:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9498.539656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9504.244189] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 9507.415581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9507.432323] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9507.588888] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9513.606829] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9518.845904] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9518.999712] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9523.932830] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9524.101259] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9528.291528] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9528.294369] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9528.322085] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9528.341458] Lustre: Skipped 1 previous similar message [ 9532.388744] Lustre: server umount lustre-OST0000 complete [ 9538.532117] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9538.545453] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9538.558321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9538.558330] Lustre: Skipped 1 previous similar message [ 9538.590450] LustreError: Skipped 1 previous similar message [ 9543.656383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9543.666980] Lustre: Skipped 2 previous similar messages [ 9548.769737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9548.782325] Lustre: Skipped 1 previous similar message [ 9550.305994] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9550.795683] Lustre: server umount lustre-MDT0000 complete [ 9553.901508] LustreError: 306795:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9553.924090] LustreError: 306795:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 9555.009577] LustreError: 305564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449395 with bad export cookie 10449832667480433364 [ 9555.010620] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9555.020054] LustreError: 305564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9555.487469] Lustre: server umount lustre-MDT0001 complete [ 9566.077793] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 11:30:05 (1788449405) [ 9568.268899] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9570.102695] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 11:30:09 (1788449409) [ 9573.742952] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 9602.806265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9602.987652] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9603.002755] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9603.161919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9607.291801] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 9615.270090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9615.324279] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9615.339879] Lustre: 311399:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9619.560951] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 9640.558910] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9640.838523] Lustre: Denying initial registration attempt from nid 192.168.201.127@tcp, specified as failover [ 9640.854239] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9640.863714] LustreError: 312853:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -99 [ 9640.885425] LustreError: 312853:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9640.899401] LustreError: 312853:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9640.942447] Lustre: server umount lustre-OST0000 complete [ 9640.946427] LustreError: 312853:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9651.681649] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9651.686755] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9651.695477] Lustre: Skipped 1 previous similar message [ 9651.698559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9653.571280] Lustre: server umount lustre-MDT0000 complete [ 9657.267465] LustreError: 311392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449497 with bad export cookie 10449832667480434239 [ 9657.280751] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9657.284240] LustreError: 311392:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9657.591301] Lustre: server umount lustre-MDT0001 complete [ 9667.705287] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 9670.155548] Key type lgssc unregistered [ 9670.390787] LNet: 313972:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9670.395586] LNetError: 313972:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9670.410964] LNet: Removed LNI 192.168.201.127@tcp [ 9671.089369] Key type .llcrypt unregistered [ 9671.091787] Key type ._llcrypt unregistered [ 9682.328104] Key type ._llcrypt registered [ 9682.332481] Key type .llcrypt registered [ 9682.444238] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 9683.886636] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9683.901197] alg: No test for adler32 (adler32-zlib) [ 9685.301899] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 9685.668434] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 9687.407189] Key type lgssc registered [ 9688.991365] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9741.818117] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 9753.259816] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9753.273716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9754.547568] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9754.566801] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9754.793963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9760.010576] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 9768.995536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9769.081771] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9769.121954] Lustre: 316732:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9769.352643] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9773.732377] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 9781.032156] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9785.719968] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9804.336344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9804.536679] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9804.699931] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9809.871865] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 9817.108604] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9822.690056] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9822.700273] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9822.718278] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9823.215135] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9823.245060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9823.250651] Lustre: Skipped 1 previous similar message [ 9825.672697] Lustre: server umount lustre-OST0000 complete [ 9830.880356] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9830.886377] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9830.896768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9833.441598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9833.443903] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9835.541337] Lustre: server umount lustre-MDT0000 complete [ 9839.627110] LustreError: 316725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449679 with bad export cookie 6455074157956346619 [ 9839.635481] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9839.644617] LustreError: 316725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9840.198957] Lustre: server umount lustre-MDT0001 complete [ 9851.961542] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [ 9854.700953] Key type lgssc unregistered [ 9855.062127] LNet: 320306:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9855.072709] LNetError: 320306:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9855.097247] LNet: Removed LNI 192.168.201.127@tcp [ 9856.233544] Key type .llcrypt unregistered [ 9856.235805] Key type ._llcrypt unregistered [ 9892.768436] Key type ._llcrypt registered [ 9892.770773] Key type .llcrypt registered [ 9892.857186] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [ 9894.267719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9894.284052] alg: No test for adler32 (adler32-zlib) [ 9895.645699] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 9895.910492] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [ 9897.623163] Key type lgssc registered [ 9899.138971] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9908.754532] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9908.769681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9910.478768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9915.551121] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 9919.973218] LustreError: 321723:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9920.004477] LustreError: 321723:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9925.090426] LustreError: 321724:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9925.652753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9925.913286] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9930.570857] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 9938.904643] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9945.441903] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9956.329666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9956.719962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9964.727097] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [ 9974.628749] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9990.630540] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9990.634052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9990.658456] Lustre: Skipped 2 previous similar messages [ 9990.673769] Lustre: Skipped 2 previous similar messages [ 9991.227727] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.27@tcp (stopping) [ 9995.763391] LustreError: 321724:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9995.785710] LustreError: 321724:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9995.914978] Lustre: server umount lustre-MDT0000 complete [10000.263563] LustreError: 321704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449840 with bad export cookie 11393220586582440745 [10000.265537] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10000.299585] LustreError: 321704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10000.866872] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10000.868416] LustreError: 322956:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10000.878420] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10000.915386] LustreError: 322956:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10004.497445] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.27@tcp (stopping) [10004.506783] Lustre: Skipped 1 previous similar message [10005.523369] LustreError: 321720:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10005.572244] LustreError: 321720:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10006.665691] Lustre: server umount lustre-MDT0001 complete [10013.734892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10022.301370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10025.442643] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x9e1cd9c276ea18aa [10025.455643] Lustre: MGC192.168.201.127@tcp: Connection restored to 0@lo (at 0@lo) [10025.836533] LustreError: 323558:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10025.854286] LustreError: 323558:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10025.925030] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10025.981739] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10030.683841] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [10035.117136] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10039.403475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10039.768881] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10044.661650] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [10044.905604] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10055.390701] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10057.727637] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10065.004402] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10067.063312] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10072.048555] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10072.060977] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10072.074931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10072.081166] Lustre: Skipped 2 previous similar messages [10075.618497] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10075.630859] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10080.742518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10080.752528] Lustre: Skipped 3 previous similar messages [10086.368390] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10086.506656] Lustre: server umount lustre-OST0000 complete [10090.981040] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10090.981624] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10091.013763] Lustre: Skipped 1 previous similar message [10096.372991] Lustre: server umount lustre-MDT0000 complete [10100.340515] LustreError: 321705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449940 with bad export cookie 11393220586582448298 [10100.345783] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10100.349520] LustreError: 321705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10100.826190] Lustre: server umount lustre-MDT0001 complete [10112.605390] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [10115.283299] Key type lgssc unregistered [10115.619262] LNet: 328010:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10115.627342] LNetError: 328010:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10116.650640] LNet: Removed LNI 192.168.201.127@tcp [10117.718312] Key type .llcrypt unregistered [10117.730811] Key type ._llcrypt unregistered [10146.134342] Key type ._llcrypt registered [10146.139479] Key type .llcrypt registered [10146.280846] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [10147.662032] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10147.745065] alg: No test for adler32 (adler32-zlib) [10148.859643] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [10149.191755] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [10150.851324] Key type lgssc registered [10152.049847] Lustre: Echo OBD driver; http://www.lustre.org/ [10160.677457] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10160.707393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10162.283554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10167.333518] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [10171.880348] LustreError: 329427:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10171.892782] LustreError: 329427:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10176.642090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10176.996589] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10177.305747] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10182.054625] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [10190.957517] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10198.293201] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10202.592689] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10202.599718] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10202.624899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10203.124652] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10204.194104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10204.196865] Lustre: Skipped 2 previous similar messages [10208.233827] Lustre: server umount lustre-MDT0000 complete [10209.253659] LustreError: 330431:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10209.284826] LustreError: 330431:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10211.727403] LustreError: 329408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450051 with bad export cookie 8722475377423900526 [10211.749763] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10212.104393] Lustre: server umount lustre-MDT0001 complete [10226.642768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10227.033713] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10227.070385] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10227.402446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10232.505992] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [10241.715680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10241.832531] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10241.857438] Lustre: 332016:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10241.945740] LustreError: 332028:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10242.115133] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10246.900686] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [10254.420066] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10261.418700] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10271.462105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10271.760025] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10271.780106] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10271.800460] LustreError: 333901:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [10271.808444] LustreError: 333901:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [10271.815206] LustreError: 333901:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10271.916058] Lustre: server umount lustre-OST0000 complete [10271.919271] LustreError: 333901:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10282.342859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10282.608527] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10282.792975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10288.823841] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10289.730874] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [10297.895222] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10305.619781] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10305.705031] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10305.785434] Lustre: lustre-OST0001: new disk, initializing [10305.788774] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10305.793563] Lustre: 335443:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10305.876098] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10311.343783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10311.366030] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10311.488802] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10312.425420] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [10320.273752] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10327.009404] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10327.025967] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10327.059702] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10332.135559] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10332.139094] Lustre: Skipped 1 previous similar message [10337.248339] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10337.420389] Lustre: server umount lustre-OST0001 complete [10342.382159] LustreError: 334368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10342.386111] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10342.392309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10342.392318] Lustre: Skipped 3 previous similar messages [10342.405836] LustreError: 334368:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10342.423074] Lustre: Skipped 2 previous similar messages [10347.494353] LustreError: 334365:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10347.525743] LustreError: 334365:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10355.167381] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10355.304933] Lustre: server umount lustre-OST0000 complete [10359.777497] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10359.783520] LustreError: Skipped 1 previous similar message [10359.786614] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10359.810367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10359.820351] Lustre: Skipped 5 previous similar messages [10364.840964] Lustre: server umount lustre-MDT0000 complete [10367.974290] LustreError: 335991:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10367.990261] LustreError: 335991:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10368.651329] LustreError: 332756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450208 with bad export cookie 8722475377423901072 [10368.657648] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10368.661574] LustreError: 332756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [10368.907171] Lustre: server umount lustre-MDT0001 complete [10379.740252] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [10382.587882] Key type lgssc unregistered [10382.912128] LNet: 337402:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10382.929572] LNetError: 337402:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10382.989622] LNet: Removed LNI 192.168.201.127@tcp [10384.038250] Key type .llcrypt unregistered [10384.039476] Key type ._llcrypt unregistered [10458.825757] Key type ._llcrypt registered [10458.831268] Key type .llcrypt registered [10458.926248] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [10474.898661] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [10476.493845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10476.555628] alg: No test for adler32 (adler32-zlib) [10477.563450] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [10477.875913] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [10479.592055] Key type lgssc registered [10480.564892] Lustre: Echo OBD driver; http://www.lustre.org/ [10520.796982] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10520.821342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10522.169991] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10522.239566] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10522.314315] Lustre: lustre-MDT0000: new disk, initializing [10522.379739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10522.402469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10526.449693] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [10536.749173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10536.885792] Lustre: 342331:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10536.932465] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10536.935890] Lustre: Skipped 1 previous similar message [10536.996208] Lustre: lustre-MDT0001: new disk, initializing [10537.083129] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10537.111026] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10537.128428] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10541.419433] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [10551.091659] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10557.549486] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10564.134733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10564.531348] Lustre: lustre-OST0000: new disk, initializing [10564.536301] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10564.541407] Lustre: 344298:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10564.621085] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10570.811615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10570.824670] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10570.942384] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10571.891226] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [10582.164161] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10586.847386] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10587.082165] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10591.103868] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10591.338406] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10596.833479] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10596.834325] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10596.850222] LustreError: Skipped 1 previous similar message [10596.851351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10596.888558] Lustre: Skipped 1 previous similar message [10599.400971] Lustre: server umount lustre-OST0000 complete [10603.488752] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10603.500508] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10603.519360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10603.529941] Lustre: Skipped 1 previous similar message [10607.076145] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10607.082703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10607.099349] Lustre: Skipped 1 previous similar message [10609.727652] Lustre: server umount lustre-MDT0000 complete [10612.736674] LustreError: 342323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450452 with bad export cookie 17320381730136351340 [10612.739836] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10612.746625] LustreError: 342323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10613.048720] Lustre: server umount lustre-MDT0001 complete [10620.489680] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 11:47:40 (1788450460) [10627.439077] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 11:47:47 (1788450467) [10635.375762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10635.806573] LustreError: 347054:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10635.829696] LustreError: 347054:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10635.871701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10640.500476] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [10640.865450] LustreError: 347055:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10644.969711] LustreError: 347054:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10649.380118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10649.903513] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10654.465165] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [10662.631254] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10668.256327] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10674.072942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10674.409980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10679.741180] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [10687.386678] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10695.816487] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10698.354794] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10701.227816] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10703.594563] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10706.313950] Lustre: Failing over lustre-MDT0000 [10706.401220] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10706.408542] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10706.411000] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10706.447658] Lustre: Skipped 2 previous similar messages [10706.752430] Lustre: server umount lustre-MDT0000 complete [10710.589686] LustreError: 349681:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10715.655804] LustreError: 347050:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10715.684166] LustreError: 347050:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10725.894425] LustreError: 347049:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10725.918485] LustreError: 347049:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10726.879161] Lustre: 340553:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788450551/real 1788450551] req@000000005d76834b x1875326080718976/t0(0) o400->MGC192.168.201.127@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788450567 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10726.896189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10726.902917] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10737.121888] LustreError: 340552:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000007efc1458 x1875326080723840/t0(0) o250->MGC192.168.201.127@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10737.436611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10737.497417] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10741.278680] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10741.530756] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [10742.779329] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10742.809750] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10742.856665] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10751.052735] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10752.596955] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10756.371732] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10758.615747] Lustre: Failing over lustre-MDT0000 [10758.896824] Lustre: server umount lustre-MDT0000 complete [10761.767189] LustreError: 347051:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10761.777488] LustreError: 347051:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [10763.233751] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10763.238845] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10763.265956] Lustre: Skipped 1 previous similar message [10777.246107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10777.348299] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10777.585140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10777.630919] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10781.728354] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [10782.230689] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10782.714143] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10782.727962] Lustre: Skipped 2 previous similar messages [10782.785710] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10782.836745] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10791.154123] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10793.035634] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10803.189290] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10803.193097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10803.228573] Lustre: Skipped 3 previous similar messages [10803.241966] Lustre: Skipped 3 previous similar messages [10805.625788] Lustre: server umount lustre-MDT0000 complete [10808.288971] LustreError: 347051:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10808.310317] LustreError: 347051:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [10809.677123] LustreError: 347035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450649 with bad export cookie 17320381730136354049 [10809.683568] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10809.695360] LustreError: 347035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10810.123570] Lustre: server umount lustre-MDT0001 complete [10824.176270] Lustre: server umount lustre-OST0000 complete [10839.020319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10844.077874] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [10853.511583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10853.825979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10853.833360] Lustre: Skipped 1 previous similar message [10858.786710] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [10866.254753] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10872.668334] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10879.836714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10884.295760] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10886.596044] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [10895.752650] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10909.152464] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10909.155294] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10909.166132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10909.171016] Lustre: Skipped 1 previous similar message [10909.183773] Lustre: Skipped 2 previous similar messages [10913.066099] Lustre: server umount lustre-OST0000 complete [10922.700612] Lustre: server umount lustre-MDT0000 complete [10924.513480] LustreError: 354575:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10924.541102] LustreError: 354575:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10926.838137] LustreError: 353377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450767 with bad export cookie 17320381730136354798 [10926.855701] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10926.858350] LustreError: 353377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10927.291928] Lustre: server umount lustre-MDT0001 complete [10937.748487] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [10940.609644] Key type lgssc unregistered [10940.920509] LNet: 356930:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10940.930188] LNetError: 356930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10940.950411] LNet: Removed LNI 192.168.201.127@tcp [10941.756910] Key type .llcrypt unregistered [10941.758521] Key type ._llcrypt unregistered [10969.102566] Key type ._llcrypt registered [10969.104855] Key type .llcrypt registered [10969.193194] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [10970.740818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10970.775561] alg: No test for adler32 (adler32-zlib) [10971.845709] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [10972.034840] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [10973.695731] Key type lgssc registered [10974.845703] Lustre: Echo OBD driver; http://www.lustre.org/ [10982.921379] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10982.937836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10984.406961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10988.720542] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [10994.663429] LustreError: 358446:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10994.678639] LustreError: 358446:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10997.603086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10997.981303] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11001.941875] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11010.053916] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11014.989954] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11021.111955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11021.461782] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11027.840541] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11029.689870] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11036.607100] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11048.295489] Lustre: server umount lustre-MDT0000 complete [11049.446358] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11049.457478] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11049.479405] LustreError: 358447:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11050.478720] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11050.481840] LustreError: 359688:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11050.535341] LustreError: 359688:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11052.552339] LustreError: 358427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450892 with bad export cookie 15547647908520035873 [11052.554153] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11052.557924] LustreError: 358427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11052.827563] Lustre: server umount lustre-MDT0001 complete [11067.473425] Lustre: server umount lustre-OST0000 complete [11080.102900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11080.526626] LustreError: 362161:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11080.588535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11084.461583] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11085.799289] LustreError: 362162:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11092.249168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11092.629706] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11097.049975] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11104.710417] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11111.009709] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11117.741170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11118.027013] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11123.687358] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11125.245197] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11130.798924] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11143.649359] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11143.659688] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11143.685195] Lustre: Skipped 1 previous similar message [11143.695439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11147.744430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11147.751213] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11147.768834] Lustre: Skipped 1 previous similar message [11149.252991] Lustre: server umount lustre-MDT0000 complete [11152.418164] LustreError: 362141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450992 with bad export cookie 15547647908520036986 [11152.430835] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11152.438363] LustreError: 362141:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11152.809416] Lustre: server umount lustre-MDT0001 complete [11166.263144] Lustre: server umount lustre-OST0000 complete [11181.823122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11182.175226] LustreError: 366069:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11182.186526] LustreError: 366069:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11182.234758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11185.889031] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11194.061979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11198.137257] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11205.279962] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11210.867147] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11216.825504] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11217.195883] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11217.205028] Lustre: Skipped 1 previous similar message [11223.966132] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11227.451059] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11232.037551] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11239.904252] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11239.907256] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11239.913179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11239.915562] Lustre: Skipped 2 previous similar messages [11242.977280] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11242.988405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11245.037188] Lustre: server umount lustre-OST0000 complete [11250.656699] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11250.673244] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11250.697803] Lustre: Skipped 1 previous similar message [11250.701993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11250.712634] Lustre: Skipped 1 previous similar message [11256.314414] Lustre: server umount lustre-MDT0000 complete [11258.338028] LustreError: 366070:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11258.362767] LustreError: 366070:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11260.157140] LustreError: 367912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451100 with bad export cookie 15547647908520038001 [11260.158025] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11260.174508] LustreError: 367912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11260.825229] Lustre: server umount lustre-MDT0001 complete [11273.221346] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [11275.979245] Key type lgssc unregistered [11276.261826] LNet: 369600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11276.266052] LNetError: 369600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11276.278600] LNet: Removed LNI 192.168.201.127@tcp [11277.282742] Key type .llcrypt unregistered [11277.285670] Key type ._llcrypt unregistered [11303.476860] Key type ._llcrypt registered [11303.479120] Key type .llcrypt registered [11303.609919] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [11305.085713] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11305.131924] alg: No test for adler32 (adler32-zlib) [11306.178631] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11306.368264] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [11308.016462] Key type lgssc registered [11309.117431] Lustre: Echo OBD driver; http://www.lustre.org/ [11317.433527] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11317.466712] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11317.475487] LustreError: 371040:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11317.499459] LustreError: 371040:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11317.512660] LustreError: 371040:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11317.518812] LustreError: 371040:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11317.531520] LustreError: 371040:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11327.584656] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11327.591853] LustreError: 371437:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11327.609907] LustreError: 371437:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11327.614912] LustreError: 371437:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11327.620911] LustreError: 371437:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11327.627746] LustreError: 371437:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11348.623782] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [11351.054384] Key type lgssc unregistered [11351.340880] LNet: 372441:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11351.355570] LNetError: 372441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11351.373254] LNet: Removed LNI 192.168.201.127@tcp [11352.055680] Key type .llcrypt unregistered [11352.058324] Key type ._llcrypt unregistered [11369.516880] Key type ._llcrypt registered [11369.518465] Key type .llcrypt registered [11369.648518] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [11385.043277] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [11386.139451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11386.243845] alg: No test for adler32 (adler32-zlib) [11387.249163] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11387.421971] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [11389.071197] Key type lgssc registered [11389.922143] Lustre: Echo OBD driver; http://www.lustre.org/ [11426.052590] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11426.129783] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11427.494393] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11427.565640] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11427.688412] Lustre: lustre-MDT0000: new disk, initializing [11427.827716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11427.855672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11433.938945] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11445.992621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11446.113739] Lustre: 375605:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11446.197787] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11446.200558] Lustre: Skipped 1 previous similar message [11446.269330] Lustre: lustre-MDT0001: new disk, initializing [11446.395066] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11446.439926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11446.455958] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11451.254175] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11461.297976] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11467.425707] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11473.936650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11474.137540] Lustre: lustre-OST0000: new disk, initializing [11474.140808] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11474.146116] Lustre: 377571:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11474.220650] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11479.870658] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11482.762627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11482.768143] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11482.875071] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11489.639528] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11495.371857] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11495.629695] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11500.399928] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11500.614769] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11503.585921] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11503.606423] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11503.631519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11503.641176] Lustre: Skipped 1 previous similar message [11508.905545] Lustre: server umount lustre-OST0000 complete [11513.828499] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11513.845465] Lustre: Skipped 2 previous similar messages [11513.846956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11513.858569] Lustre: Skipped 1 previous similar message [11518.950512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11518.956695] Lustre: Skipped 1 previous similar message [11519.406774] Lustre: server umount lustre-MDT0000 complete [11523.395184] LustreError: 375596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451363 with bad export cookie 14393104032085696182 [11523.398410] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11523.407321] LustreError: 375596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11523.808860] Lustre: server umount lustre-MDT0001 complete [11532.448706] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11534.363513] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 12:02:53 (1788451373) [11540.473849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11540.854740] LustreError: 380032:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11540.871078] LustreError: 380032:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11540.919204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11544.642928] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11546.083850] LustreError: 380033:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11551.213122] LustreError: 380032:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11552.998698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11553.430162] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11557.080469] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11564.447459] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11570.173522] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11575.811792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11576.038464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11582.289889] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11590.202652] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11597.015235] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11597.150743] Lustre: lustre-OST0001: new disk, initializing [11597.153846] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11597.158609] Lustre: 382868:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11597.229816] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11602.824610] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11603.526451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11603.544554] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11603.663623] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11611.891884] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11618.861367] Lustre: lustre-OST0001: Not available for connect from 192.168.201.27@tcp (stopping) [11619.303133] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11619.304507] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11619.306835] LustreError: Skipped 2 previous similar messages [11619.324510] Lustre: Skipped 1 previous similar message [11621.717863] Lustre: server umount lustre-OST0001 complete [11623.943072] LustreError: 383276:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11629.539123] LustreError: 381851:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11629.555574] LustreError: 381851:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11637.731049] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11637.740313] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11637.761070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11637.768475] Lustre: Skipped 2 previous similar messages [11639.789571] LustreError: 383792:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11639.794593] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11639.814567] LustreError: 383792:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11643.547446] Lustre: server umount lustre-OST0000 complete [11650.021965] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11650.031233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11650.049485] Lustre: Skipped 2 previous similar messages [11650.059114] Lustre: Skipped 3 previous similar messages [11653.655975] Lustre: server umount lustre-MDT0000 complete [11657.425987] LustreError: 380013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451497 with bad export cookie 14393104032085697064 [11657.433461] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11657.434681] LustreError: 380013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11663.821369] Lustre: server umount lustre-MDT0001 complete [11676.864584] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [11679.925608] Key type lgssc unregistered [11680.263327] LNet: 384857:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11680.271809] LNetError: 384857:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11680.288142] LNet: Removed LNI 192.168.201.127@tcp [11681.113609] Key type .llcrypt unregistered [11681.119657] Key type ._llcrypt unregistered [11738.977687] Key type ._llcrypt registered [11738.979966] Key type .llcrypt registered [11739.078971] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [11740.744124] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11740.779970] alg: No test for adler32 (adler32-zlib) [11742.009317] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11742.396357] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [11744.087252] Key type lgssc registered [11745.230258] Lustre: Echo OBD driver; http://www.lustre.org/ [11752.166289] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11752.181259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11753.413129] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11753.443952] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11753.646572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11758.259569] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11765.883203] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11766.019228] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11766.081831] Lustre: 387654:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11766.375611] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11770.809093] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11778.316895] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11784.407456] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11786.720507] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11786.730351] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11786.752499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11787.250495] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11787.268319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11787.273579] Lustre: Skipped 1 previous similar message [11792.352559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11792.357314] Lustre: Skipped 1 previous similar message [11793.091919] Lustre: server umount lustre-MDT0000 complete [11796.486045] LustreError: 387645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451636 with bad export cookie 2293103875169803219 [11796.509396] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11796.800343] Lustre: server umount lustre-MDT0001 complete [11808.451680] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 12:07:27 (1788451647) [11816.264210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11816.767153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11821.895727] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11822.066338] LustreError: 390164:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11822.081085] LustreError: 390164:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11827.188267] LustreError: 390165:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11831.014850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11831.371703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11836.469654] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11844.741974] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11851.613245] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11859.277673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11859.591573] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11859.841985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11866.605445] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11874.790535] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11883.892675] Lustre: Permanently deactivating lustre-OST0000 [11883.928127] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11893.298607] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11895.855451] LustreError: 393199:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11895.862847] LustreError: 393199:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11895.868312] LustreError: 393199:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11899.434848] LustreError: 393247:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11899.442450] LustreError: 393247:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11907.570704] Lustre: server umount lustre-OST0000 complete [11910.117267] LustreError: 393499:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11910.123498] LustreError: 393499:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11912.161614] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11912.167607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11912.183599] Lustre: Skipped 1 previous similar message [11912.200722] Lustre: Skipped 1 previous similar message [11917.281623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11917.292202] Lustre: Skipped 1 previous similar message [11922.544655] Lustre: server umount lustre-MDT0000 complete [11926.618082] LustreError: 390145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451766 with bad export cookie 2293103875169803800 [11926.622808] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11926.626210] LustreError: 390145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11936.437140] Lustre: server umount lustre-MDT0001 complete [11945.023895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11949.588292] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11952.417357] LustreError: 394657:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11958.741959] Lustre: 394996:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.127@tcp [11960.623423] Lustre: server umount MGS complete [11972.231677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11972.635209] LustreError: 395720:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11972.647151] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11972.662205] Lustre: Skipped 1 previous similar message [11972.716462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11976.651097] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11978.216033] LustreError: 395721:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11983.337126] LustreError: 395720:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11985.262186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11985.541337] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11991.041954] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [11999.333936] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12005.385688] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12007.446736] Lustre: Permanently reactivating lustre-OST0000 [12007.474846] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12007.482619] Lustre: Skipped 2 previous similar messages [12036.575251] LustreError: 397528:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12045.016238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12045.522627] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12045.527634] Lustre: Skipped 1 previous similar message [12052.847443] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [12062.105294] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12070.531255] Lustre: server umount lustre-OST0000 complete [12077.553734] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12077.569098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12077.578343] Lustre: Skipped 1 previous similar message [12077.595690] Lustre: Skipped 3 previous similar messages [12084.778382] Lustre: server umount lustre-MDT0000 complete [12087.801311] LustreError: 395715:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12087.835328] LustreError: 395715:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12088.549479] LustreError: 395703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451928 with bad export cookie 2293103875169804850 [12088.551213] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12088.563306] LustreError: 395703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12098.370889] Lustre: server umount lustre-MDT0001 complete [12110.046813] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [12112.502514] Key type lgssc unregistered [12112.870165] LNet: 400137:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12112.877729] LNetError: 400137:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12112.890215] LNet: Removed LNI 192.168.201.127@tcp [12113.785190] Key type .llcrypt unregistered [12113.787541] Key type ._llcrypt unregistered [12133.563270] Key type ._llcrypt registered [12133.565643] Key type .llcrypt registered [12133.644796] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [12151.490846] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [12152.704388] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12152.983864] alg: No test for adler32 (adler32-zlib) [12154.069615] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12154.319448] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [12156.047286] Key type lgssc registered [12157.227577] Lustre: Echo OBD driver; http://www.lustre.org/ [12195.440875] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12195.461330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12196.805573] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12196.852866] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12196.916959] Lustre: lustre-MDT0000: new disk, initializing [12197.017710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12197.041116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12200.935497] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [12213.270569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12213.452769] Lustre: 403293:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12213.533737] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12213.543244] Lustre: Skipped 1 previous similar message [12213.685924] Lustre: lustre-MDT0001: new disk, initializing [12213.817856] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12213.866841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12213.878508] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12218.672215] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [12228.086167] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12234.641655] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12242.398644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12242.797587] Lustre: lustre-OST0000: new disk, initializing [12242.801985] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12242.812624] Lustre: 405262:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12242.896393] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12250.228281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12250.248825] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12250.460432] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12251.520827] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [12262.504476] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12268.542477] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12268.782648] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12273.631854] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12273.879650] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12276.204200] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12276.208208] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12276.210221] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12276.221202] Lustre: Skipped 1 previous similar message [12281.327134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12281.332564] Lustre: Skipped 1 previous similar message [12282.282381] Lustre: server umount lustre-OST0000 complete [12290.529809] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12290.548112] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12290.591246] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12291.558724] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12296.672562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12296.681960] Lustre: Skipped 3 previous similar messages [12301.281116] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12301.448855] Lustre: server umount lustre-MDT0000 complete [12305.735289] LustreError: 403284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452145 with bad export cookie 15078556718627846134 [12305.748763] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12305.997151] Lustre: server umount lustre-MDT0001 complete [12315.690829] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12317.747307] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 12:15:57 (1788452157) [12325.669254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12326.086173] LustreError: 407725:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12326.116342] LustreError: 407725:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12326.210769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12330.275587] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [12331.490973] LustreError: 407726:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12336.609589] LustreError: 407725:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12338.815531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12339.248226] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12344.080666] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [12352.470959] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12358.231553] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12366.023142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12366.336388] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12372.403540] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [12381.748792] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12383.435256] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12386.309854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12394.989208] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12395.012977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12395.013579] Lustre: Skipped 1 previous similar message [12395.046281] Lustre: Skipped 1 previous similar message [12399.370768] Lustre: server umount lustre-OST0000 complete [12405.221284] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12405.224231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12405.243247] Lustre: Skipped 1 previous similar message [12410.341380] LustreError: 407721:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12410.366961] LustreError: 407721:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12410.492587] Lustre: server umount lustre-MDT0000 complete [12414.154258] LustreError: 407706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452254 with bad export cookie 15078556718627847016 [12414.171753] LustreError: 407706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [12414.174130] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12414.569249] Lustre: server umount lustre-MDT0001 complete [12428.029840] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [12431.194703] Key type lgssc unregistered [12431.523950] LNet: 411451:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12431.534731] LNetError: 411451:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12431.549115] LNet: Removed LNI 192.168.201.127@tcp [12432.532411] Key type .llcrypt unregistered [12432.536060] Key type ._llcrypt unregistered [12458.867450] Key type ._llcrypt registered [12458.869953] Key type .llcrypt registered [12459.005528] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [12459.369496] Key type .llcrypt unregistered [12459.380496] Key type ._llcrypt unregistered [12478.133188] Key type ._llcrypt registered [12478.135384] Key type .llcrypt registered [12478.229482] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [12479.134385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12479.194877] alg: No test for adler32 (adler32-zlib) [12480.365243] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12480.598786] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [12482.280840] Key type lgssc registered [12483.579664] Lustre: Echo OBD driver; http://www.lustre.org/ [12493.462634] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12493.485230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12494.883955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12500.273598] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [12505.061237] LustreError: 413674:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12505.085753] LustreError: 413674:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12510.185558] LustreError: 413675:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12511.582225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12512.014533] LustreError: 413674:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12512.056503] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12517.348866] LustreError: 413675:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12517.370865] LustreError: 413675:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12520.551411] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12521.925940] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [12525.545134] LustreError: 413670:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12525.580669] LustreError: 413670:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12532.574854] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12535.779538] LustreError: 413670:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12535.797903] LustreError: 413670:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12540.517644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12541.046511] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12545.800942] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [12556.014393] Lustre: server umount lustre-OST0000 complete [12560.996975] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12561.003724] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12561.017937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12561.888970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12561.889860] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12567.009069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12567.023108] Lustre: Skipped 2 previous similar messages [12571.184462] Lustre: server umount lustre-MDT0000 complete [12572.127197] Lustre: 412985:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788452396/real 1788452396] req@0000000001c39585 x1875328179924480/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788452412 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12572.129326] LustreError: 413670:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12572.169627] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12574.581867] LustreError: 413654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452414 with bad export cookie 5017630447625594189 [12574.600460] LustreError: 413654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12574.605426] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12574.956508] Lustre: server umount lustre-MDT0001 complete [12586.927756] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [12589.527980] Key type lgssc unregistered [12589.795890] LNet: 416825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12589.802887] LNetError: 416825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12590.825767] LNet: Removed LNI 192.168.201.127@tcp [12591.846910] Key type .llcrypt unregistered [12591.851408] Key type ._llcrypt unregistered [12621.723788] Key type ._llcrypt registered [12621.725865] Key type .llcrypt registered [12621.877870] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [12623.684336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12623.720310] alg: No test for adler32 (adler32-zlib) [12625.237486] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12625.507544] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [12627.199650] Key type lgssc registered [12628.020848] Lustre: Echo OBD driver; http://www.lustre.org/ [12637.237465] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12637.270771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12652.959585] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12658.080710] LustreError: 418212:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.127@tcp: failed processing log, type 4: rc = -110 [12684.767556] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12691.486982] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [12699.845390] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12707.249624] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12707.952704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12708.021573] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12712.954582] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [12713.456753] LustreError: 418238:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12713.479448] LustreError: 418238:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12718.564801] LustreError: 418237:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12719.611332] LustreError: 419251:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12724.390715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12724.705072] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [12724.714740] Lustre: Skipped 1 previous similar message [12724.862600] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12729.599930] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [12737.833118] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12744.897935] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12753.900312] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12753.911077] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12753.939807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12754.295681] Lustre: server umount lustre-OST0000 complete [12760.049076] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12760.065172] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12760.089381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12760.840510] Lustre: server umount lustre-MDT0000 complete [12761.056526] LustreError: 419255:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12761.079819] LustreError: 419255:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12761.099393] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12764.905377] LustreError: 418219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452605 with bad export cookie 1710913648548529627 [12764.913257] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12764.925826] LustreError: 418219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12775.553510] Lustre: server umount lustre-MDT0001 complete [12787.586454] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [12790.191789] Key type lgssc unregistered [12790.545259] LNet: 421784:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12790.553560] LNetError: 421784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12790.591314] LNet: Removed LNI 192.168.201.127@tcp [12791.621863] Key type .llcrypt unregistered [12791.649223] Key type ._llcrypt unregistered [12822.676190] Key type ._llcrypt registered [12822.678331] Key type .llcrypt registered [12822.855141] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [12824.763452] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12824.809618] alg: No test for adler32 (adler32-zlib) [12826.046503] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12826.305974] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [12828.000944] Key type lgssc registered [12829.798071] Lustre: Echo OBD driver; http://www.lustre.org/ [12839.685431] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12839.722890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12841.473517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12847.140143] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [12851.693071] LustreError: 423200:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12851.727599] LustreError: 423200:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12856.811705] LustreError: 423199:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12857.152476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12857.682158] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12862.455803] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [12871.158272] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12877.395110] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12885.303175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12885.774579] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12886.847682] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12892.679632] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [12900.136801] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12907.490239] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12907.495477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12907.518155] Lustre: Skipped 1 previous similar message [12908.073062] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.27@tcp (stopping) [12908.087869] Lustre: Skipped 2 previous similar messages [12911.227640] Lustre: server umount lustre-MDT0000 complete [12912.614110] LustreError: 423194:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12912.635669] LustreError: 423194:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12913.531257] LustreError: 423180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.27@tcp arrived at 1788452753 with bad export cookie 12168602540516653043 [12915.089839] LustreError: 423182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452755 with bad export cookie 12168602540516652154 [12915.102153] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12915.111684] LustreError: 423182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12917.729945] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12917.731604] LustreError: 423194:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12917.744255] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12917.744263] Lustre: Skipped 1 previous similar message [12917.751478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12917.764994] LustreError: 423194:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12920.739032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12921.348684] Lustre: server umount lustre-OST0000 complete [12924.838825] LustreError: 423194:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12924.862564] LustreError: 423194:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12928.934731] Lustre: server umount lustre-MDT0001 complete [12941.225677] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [12943.655475] Key type lgssc unregistered [12944.022282] LNet: 426886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12944.031357] LNetError: 426886:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12945.063775] LNet: Removed LNI 192.168.201.127@tcp [12945.970570] Key type .llcrypt unregistered [12945.978437] Key type ._llcrypt unregistered [12974.769758] Key type ._llcrypt registered [12974.771956] Key type .llcrypt registered [12974.877805] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [12975.897585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12976.019163] alg: No test for adler32 (adler32-zlib) [12977.100883] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12977.311036] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [12979.055223] Key type lgssc registered [12980.031201] Lustre: Echo OBD driver; http://www.lustre.org/ [12988.819742] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12988.834187] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12990.283798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12995.247521] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [13000.684311] LustreError: 428301:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13000.758604] LustreError: 428301:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13005.795306] LustreError: 428302:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13005.921804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13006.244845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13012.280201] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [13023.812637] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13031.244882] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13039.666997] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13040.101588] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13047.869447] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [13060.324664] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13060.618737] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13067.749902] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13067.760196] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13067.835213] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13068.357826] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.27@tcp (stopping) [13068.718924] Lustre: server umount lustre-MDT0001 complete [13070.822112] LustreError: 429029:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13070.836775] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13075.941188] LustreError: 428302:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13075.967954] LustreError: 428302:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13081.056632] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13081.058341] LustreError: 428301:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13081.063518] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13081.096520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13084.969432] Lustre: server umount lustre-OST0000 complete [13088.880570] Lustre: server umount lustre-MDT0000 complete [13105.208919] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [13108.375497] Key type lgssc unregistered [13108.662339] LNet: 431990:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13108.681232] LNetError: 431990:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13108.698629] LNet: Removed LNI 192.168.201.127@tcp [13109.567123] Key type .llcrypt unregistered [13109.569275] Key type ._llcrypt unregistered [13140.660328] Key type ._llcrypt registered [13140.662542] Key type .llcrypt registered [13140.799894] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [13161.392527] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [13162.739429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13162.823980] alg: No test for adler32 (adler32-zlib) [13164.215761] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [13164.451880] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [13166.119839] Key type lgssc registered [13167.619526] Lustre: Echo OBD driver; http://www.lustre.org/ [13212.541746] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [13233.203863] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13233.250780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13234.749782] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13234.803443] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13234.945253] Lustre: lustre-MDT0000: new disk, initializing [13235.081989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13235.113630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13240.995722] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [13258.809449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13259.184896] Lustre: lustre-OST0000: new disk, initializing [13259.190306] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13259.198482] Lustre: Skipped 1 previous similar message [13259.211233] Lustre: 437024:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13259.351318] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13266.774763] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [13267.034944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13267.048790] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13267.103994] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13277.693251] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13298.556115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13298.838646] Lustre: 436006:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13298.940236] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13299.050430] Lustre: lustre-MDT0001: new disk, initializing [13299.231615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13299.338931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13299.366033] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13305.723803] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [13317.239882] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13317.423794] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13323.596561] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13324.117452] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13333.473086] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13333.497182] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13334.079211] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.27@tcp (stopping) [13334.093731] Lustre: Skipped 1 previous similar message [13337.189099] Lustre: server umount lustre-MDT0001 complete [13343.713990] LustreError: 436017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13343.769964] LustreError: 436017:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [13348.838680] LustreError: 436013:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13348.853269] LustreError: 436013:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13353.956532] LustreError: 436017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13353.974452] LustreError: 436017:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13359.073485] LustreError: 436018:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13359.079917] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13359.128936] Lustre: Skipped 1 previous similar message [13359.148385] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13363.727434] Lustre: server umount lustre-OST0000 complete [13364.195974] LustreError: 436017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13374.347123] Lustre: server umount lustre-MDT0000 complete [13391.376599] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [13394.389556] Key type lgssc unregistered [13394.783977] LNet: 440775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13394.791140] LNetError: 440775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13394.803324] LNet: Removed LNI 192.168.201.127@tcp [13395.678816] Key type .llcrypt unregistered [13395.681351] Key type ._llcrypt unregistered [13444.800840] Key type ._llcrypt registered [13444.802346] Key type .llcrypt registered [13444.947951] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [13445.233438] Key type .llcrypt unregistered [13445.236794] Key type ._llcrypt unregistered [13464.916384] Key type ._llcrypt registered [13464.918482] Key type .llcrypt registered [13465.006313] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [13466.516175] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13466.532656] alg: No test for adler32 (adler32-zlib) [13467.681128] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [13467.914529] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [13469.560919] Key type lgssc registered [13470.863532] Lustre: Echo OBD driver; http://www.lustre.org/ [13501.125700] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13501.165650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13502.568617] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13502.622267] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13502.778522] Lustre: lustre-MDT0000: new disk, initializing [13502.898063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13502.922936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13509.385861] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [13520.949164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13521.118463] Lustre: 444146:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13521.172616] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13521.177336] Lustre: Skipped 1 previous similar message [13521.262686] Lustre: lustre-MDT0001: new disk, initializing [13521.354041] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13521.380825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13521.394158] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13526.435856] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [13538.205834] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13544.677475] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13552.233713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13552.469262] Lustre: lustre-OST0000: new disk, initializing [13552.474465] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13552.480061] Lustre: 446111:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13552.577830] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13559.511750] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [13560.407844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13560.417984] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13560.612964] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13571.204600] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13581.281690] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13581.291808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13581.297125] Lustre: Skipped 2 previous similar messages [13581.313020] Lustre: Skipped 2 previous similar messages [13585.914349] Lustre: server umount lustre-MDT0000 complete [13590.349150] LustreError: 444137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453430 with bad export cookie 12910369774228997418 [13590.352084] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13590.358396] LustreError: 444137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13590.864178] Lustre: server umount lustre-MDT0001 complete [13606.022307] Lustre: server umount lustre-OST0000 complete [13620.800588] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 12:37:39 (1788453459) [13631.172683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13631.784761] LustreError: 448145:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13631.803629] LustreError: 448145:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13631.866030] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13636.405398] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [13637.089406] LustreError: 448146:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13641.192634] LustreError: 448145:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13645.854635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13646.252920] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13650.813636] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [13658.884274] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13665.426356] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13686.638302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13687.014867] Lustre: Found index 0 for lustre-OST0000, updating log [13687.379722] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13695.010963] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [13702.783470] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13704.620164] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13712.356855] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13712.377145] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13712.404773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13713.513372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13713.535939] Lustre: Skipped 2 previous similar messages [13717.762241] Lustre: server umount lustre-OST0000 complete [13735.397900] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13735.415225] Lustre: Skipped 1 previous similar message [13735.426540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13735.438101] Lustre: Skipped 1 previous similar message [13740.513832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13740.526977] Lustre: Skipped 1 previous similar message [13741.537370] LustreError: 448451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13741.547261] Lustre: server umount lustre-MDT0000 complete [13741.557656] LustreError: 448451:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13746.579295] LustreError: 448127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453586 with bad export cookie 12910369774228998811 [13746.590408] LustreError: 448127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13746.590894] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13747.025587] Lustre: server umount lustre-MDT0001 complete [13756.703526] LNet: 1 local NIs in recovery (showing 1): 192.168.201.127@tcp [13757.989091] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 12:39:57 (1788453597) [14385.884347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14386.526381] LustreError: 454554:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14386.538848] LustreError: 454554:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14386.585195] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14391.783220] LustreError: 454555:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14392.154221] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [14395.884088] LustreError: 454554:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14401.531054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14402.108738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14406.840493] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [14415.647654] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14422.406086] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14443.577599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14443.898415] Lustre: Found index 0 for lustre-OST0000, updating log [14444.183038] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14448.935186] LNet: 1 local NIs in recovery (showing 1): 192.168.201.127@tcp [14451.176982] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [14459.003451] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14462.177476] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14462.207997] Lustre: Skipped 1 previous similar message [14462.210655] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14467.386397] Lustre: server umount lustre-OST0000 complete [14472.673651] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14472.686978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14472.691139] Lustre: Skipped 1 previous similar message [14474.721537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14474.729791] Lustre: Skipped 3 previous similar messages [14478.821524] LustreError: 454549:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14478.855870] LustreError: 454549:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14480.006181] Lustre: server umount lustre-MDT0000 complete [14482.913806] LustreError: 455799:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14482.947277] LustreError: 455799:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [14484.207803] LustreError: 454535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454324 with bad export cookie 12910369774228999896 [14484.214425] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14484.221491] LustreError: 454535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14484.617436] Lustre: server umount lustre-MDT0001 complete [14512.418849] LNet: 1 local NIs in recovery (showing 1): 192.168.201.127@tcp [14519.467784] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [14531.062736] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [14573.732647] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 12:53:33 (1788454413) [14596.057547] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [14598.616963] Key type lgssc unregistered [14598.873916] LNet: 461167:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14598.880046] LNetError: 461167:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14706.407736] LNet: Removed LNI 192.168.201.127@tcp [14707.630730] Key type .llcrypt unregistered [14707.632965] Key type ._llcrypt unregistered [14725.973460] Key type ._llcrypt registered [14725.978393] Key type .llcrypt registered [14726.099872] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [14727.434794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14727.562790] alg: No test for adler32 (adler32-zlib) [14728.561525] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [14728.756320] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [14730.399478] Key type lgssc registered [14731.484089] Lustre: Echo OBD driver; http://www.lustre.org/ [14736.112605] LNet: Added LNI 192.168.201.127@tcp100 [8/256/0/180] [14737.021049] LNet: Added LNI 192.168.201.127@tcp101 [8/256/0/180] [14737.838146] LNet: Added LNI 192.168.201.127@tcp102 [8/256/0/180] [14738.839424] LNet: Added LNI 192.168.201.127@tcp103 [8/256/0/180] [14739.845742] LNet: Added LNI 192.168.201.127@tcp104 [8/256/0/180] [14740.845989] LNet: Added LNI 192.168.201.127@tcp105 [8/256/0/180] [14741.847281] LNet: Added LNI 192.168.201.127@tcp106 [8/256/0/180] [14742.715602] LNet: Added LNI 192.168.201.127@tcp107 [8/256/0/180] [14743.642256] LNet: Added LNI 192.168.201.127@tcp108 [8/256/0/180] [14744.570864] LNet: Added LNI 192.168.201.127@tcp109 [8/256/0/180] [14745.393699] LNet: Added LNI 192.168.201.127@tcp110 [8/256/0/180] [14746.347855] LNet: Added LNI 192.168.201.127@tcp111 [8/256/0/180] [14747.363675] LNet: Added LNI 192.168.201.127@tcp112 [8/256/0/180] [14748.345221] LNet: Added LNI 192.168.201.127@tcp113 [8/256/0/180] [14749.542622] LNet: Added LNI 192.168.201.127@tcp114 [8/256/0/180] [14750.418880] LNet: Added LNI 192.168.201.127@tcp115 [8/256/0/180] [14751.287582] LNet: Added LNI 192.168.201.127@tcp116 [8/256/0/180] [14752.195838] LNet: Added LNI 192.168.201.127@tcp117 [8/256/0/180] [14753.129134] LNet: Added LNI 192.168.201.127@tcp118 [8/256/0/180] [14754.132910] LNet: Added LNI 192.168.201.127@tcp119 [8/256/0/180] [14755.153059] LNet: Added LNI 192.168.201.127@tcp120 [8/256/0/180] [14756.172457] LNet: Added LNI 192.168.201.127@tcp121 [8/256/0/180] [14757.177509] LNet: Added LNI 192.168.201.127@tcp122 [8/256/0/180] [14758.257934] LNet: Added LNI 192.168.201.127@tcp123 [8/256/0/180] [14759.245055] LNet: Added LNI 192.168.201.127@tcp124 [8/256/0/180] [14760.171582] LNet: Added LNI 192.168.201.127@tcp125 [8/256/0/180] [14761.010342] LNet: Added LNI 192.168.201.127@tcp126 [8/256/0/180] [14761.998027] LNet: Added LNI 192.168.201.127@tcp127 [8/256/0/180] [14762.683520] LNet: Added LNI 192.168.201.127@tcp128 [8/256/0/180] [14763.741481] LNet: Added LNI 192.168.201.127@tcp129 [8/256/0/180] [14764.715422] LNet: Added LNI 192.168.201.127@tcp130 [8/256/0/180] [14765.615509] LNet: Added LNI 192.168.201.127@tcp131 [8/256/0/180] [14766.437891] LNet: Added LNI 192.168.201.127@tcp132 [8/256/0/180] [14767.316600] LNet: Added LNI 192.168.201.127@tcp133 [8/256/0/180] [14768.122689] LNet: Added LNI 192.168.201.127@tcp134 [8/256/0/180] [14768.933973] LNet: Added LNI 192.168.201.127@tcp135 [8/256/0/180] [14775.365941] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14775.422163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14776.604208] Lustre: 464199:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.127@tcp: can't add NID 192.168.201.127@tcp130: rc = -75 [14776.681383] LNetError: 461658:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.127 [14776.698800] LNetError: Protocol error connecting to 192.168.201.127@tcp119 at host 192.168.201.127:988: is it running a compatible version of Lustre? [14776.890053] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14776.982562] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14777.139483] Lustre: lustre-MDT0000: new disk, initializing [14777.207510] Lustre: 464253:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.127@tcp: can't add NID 192.168.201.127@tcp130: rc = -75 [14777.218556] Lustre: 464253:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14777.250474] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14777.287777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14777.696113] LNetError: 461657:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.127 [14777.712664] LNetError: Protocol error connecting to 192.168.201.127@tcp105 at host 192.168.201.127:988: is it running a compatible version of Lustre? [14782.115334] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [14793.273454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14793.490367] Lustre: 464215:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14793.569619] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14793.575843] Lustre: Skipped 1 previous similar message [14793.679454] Lustre: lustre-MDT0001: new disk, initializing [14793.718885] Lustre: 465020:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.127@tcp: can't add NID 192.168.201.127@tcp130: rc = -75 [14793.733577] Lustre: 465020:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14793.871526] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14793.937146] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14793.950176] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14799.021774] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [14801.439683] Lustre: 465383:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.127@tcp: can't add NID 192.168.201.127@tcp130: rc = -75 [14801.450194] Lustre: 465383:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14809.620890] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14812.706946] LNetError: 464222:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.127 [14812.721654] LNetError: Protocol error connecting to 192.168.201.127@tcp105 at host 192.168.201.127:988: is it running a compatible version of Lustre? [14816.543686] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14816.801866] LNetError: 461655:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.127 [14816.822428] LNetError: Protocol error connecting to 192.168.201.127@tcp119 at host 192.168.201.127:988: is it running a compatible version of Lustre? [14824.002173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14824.357359] Lustre: lustre-OST0000: new disk, initializing [14824.360870] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14824.365966] Lustre: 466191:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14824.491333] Lustre: 466196:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.127@tcp: can't add NID 192.168.201.127@tcp130: rc = -75 [14824.499265] Lustre: 466196:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14824.576063] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14826.017797] LNetError: 464219:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.127 [14826.040113] LNetError: Protocol error connecting to 192.168.201.127@tcp105 at host 192.168.201.127:988: is it running a compatible version of Lustre? [14831.189338] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [14831.682080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14831.696139] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14831.838154] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14838.307658] LNetError: 461658:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.127 [14838.318306] LNetError: 461658:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [14838.324418] LNetError: Protocol error connecting to 192.168.201.127@tcp105 at host 192.168.201.127:988: is it running a compatible version of Lustre? [14838.331451] LNetError: Skipped 1 previous similar message [14841.545392] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14852.583089] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14852.586525] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14852.608835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14852.628843] Lustre: Skipped 1 previous similar message [14855.712367] LNetError: 461656:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.127 [14855.725284] LNetError: 461656:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [14855.736639] LNetError: Protocol error connecting to 192.168.201.127@tcp119 at host 192.168.201.127:988: is it running a compatible version of Lustre? [14855.758235] LNetError: Skipped 1 previous similar message [14857.052849] Lustre: server umount lustre-OST0000 complete [14862.816286] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14862.821217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14862.833513] Lustre: Skipped 2 previous similar messages [14862.844187] Lustre: Skipped 1 previous similar message [14867.331226] Lustre: server umount lustre-MDT0000 complete [14871.604643] LustreError: 464209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788454711 with bad export cookie 16818080798702214106 [14871.630675] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14872.261225] Lustre: server umount lustre-MDT0001 complete [14883.237975] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [14885.543171] Key type lgssc unregistered [14885.941259] LNet: 468104:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14885.948922] LNetError: 468104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14885.955271] LNet: Removed LNI 192.168.201.127@tcp [14887.007262] LNet: Removed LNI 192.168.201.127@tcp100 [14887.010735] LNetError: 468104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14887.016443] LNetError: 468104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14887.023555] LNet: Removed LNI 192.168.201.127@tcp101 [14888.032690] LNet: Removed LNI 192.168.201.127@tcp102 [14888.038893] LNetError: 468104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14888.048871] LNetError: 468104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14888.052890] LNet: Removed LNI 192.168.201.127@tcp103 [14889.120433] LNet: Removed LNI 192.168.201.127@tcp104 [14890.143324] LNet: Removed LNI 192.168.201.127@tcp105 [14890.145837] LNetError: 468104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14890.151379] LNetError: 468104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14890.158556] LNet: Removed LNI 192.168.201.127@tcp106 [14891.168315] LNet: Removed LNI 192.168.201.127@tcp107 [14892.191775] LNet: Removed LNI 192.168.201.127@tcp108 [14893.215190] LNet: Removed LNI 192.168.201.127@tcp109 [14894.240714] LNet: Removed LNI 192.168.201.127@tcp110 [14894.243428] LNetError: 468104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14894.252614] LNetError: 468104:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14894.258429] LNet: Removed LNI 192.168.201.127@tcp111 [14895.263250] LNet: Removed LNI 192.168.201.127@tcp112 [14895.274598] LNet: Removed LNI 192.168.201.127@tcp113 [14896.351169] LNet: Removed LNI 192.168.201.127@tcp114 [14897.375557] LNet: Removed LNI 192.168.201.127@tcp115 [14898.401856] LNet: Removed LNI 192.168.201.127@tcp116 [14899.424084] LNet: Removed LNI 192.168.201.127@tcp117 [14900.447206] LNet: Removed LNI 192.168.201.127@tcp118 [14901.471214] LNet: Removed LNI 192.168.201.127@tcp119 [14901.473392] LNet: Removed LNI 192.168.201.127@tcp120 [14901.477512] LNet: Removed LNI 192.168.201.127@tcp121 [14901.480362] LNet: Removed LNI 192.168.201.127@tcp122 [14901.483222] LNet: Removed LNI 192.168.201.127@tcp123 [14901.487964] LNet: Removed LNI 192.168.201.127@tcp124 [14901.491967] LNet: Removed LNI 192.168.201.127@tcp125 [14901.494831] LNet: Removed LNI 192.168.201.127@tcp126 [14901.498772] LNet: Removed LNI 192.168.201.127@tcp127 [14901.504225] LNet: Removed LNI 192.168.201.127@tcp128 [14901.510217] LNet: Removed LNI 192.168.201.127@tcp129 [14901.515170] LNet: Removed LNI 192.168.201.127@tcp130 [14901.516755] LNet: Removed LNI 192.168.201.127@tcp131 [14901.522486] LNet: Removed LNI 192.168.201.127@tcp132 [14901.525949] LNet: Removed LNI 192.168.201.127@tcp133 [14901.530659] LNet: Removed LNI 192.168.201.127@tcp134 [14901.546270] LNet: Removed LNI 192.168.201.127@tcp135 [14902.354727] Key type .llcrypt unregistered [14902.367059] Key type ._llcrypt unregistered [14927.163317] Key type ._llcrypt registered [14927.168750] Key type .llcrypt registered [14927.376902] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [14946.847254] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [14948.405738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14948.798494] alg: No test for adler32 (adler32-zlib) [14950.042755] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [14950.439054] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [14952.215536] Key type lgssc registered [14953.285471] Lustre: Echo OBD driver; http://www.lustre.org/ [14991.833683] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 13:00:30 (1788454830) [15011.327897] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [15014.923669] Key type lgssc unregistered [15015.206292] LNet: 472187:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15015.214462] LNetError: 472187:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15015.230545] LNet: Removed LNI 192.168.201.127@tcp [15016.066354] Key type .llcrypt unregistered [15016.068586] Key type ._llcrypt unregistered [15034.166471] Key type ._llcrypt registered [15034.172608] Key type .llcrypt registered [15034.367967] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [15035.617074] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15035.859228] alg: No test for adler32 (adler32-zlib) [15037.000989] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [15037.281538] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [15038.959888] Key type lgssc registered [15039.900459] Lustre: Echo OBD driver; http://www.lustre.org/ [15049.151143] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15049.186407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15050.573966] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15050.654886] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15050.771289] Lustre: lustre-MDT0000: new disk, initializing [15050.936561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15050.968839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15056.263360] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [15068.551379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15068.753778] Lustre: 473407:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15068.816139] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15068.821198] Lustre: Skipped 1 previous similar message [15068.909635] Lustre: lustre-MDT0001: new disk, initializing [15069.018305] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15069.060975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15069.079696] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15076.313814] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [15090.259520] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15099.123897] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15108.290259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15108.551933] Lustre: lustre-OST0000: new disk, initializing [15108.556704] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15108.562391] Lustre: 475378:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15108.640265] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15115.424460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15115.443097] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15115.687519] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15118.356349] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [15129.979192] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15134.748562] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing lnet_if_list [15141.057762] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15141.092617] LNet: Added LNI 192.168.201.127@tcp10 [8/256/0/180] [15148.266071] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15148.298204] LNet: Added LNI 192.168.201.127@tcp11 [8/256/0/180] [15153.243604] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15153.282536] LNet: Added LNI 192.168.201.127@tcp12 [8/256/0/180] [15158.699990] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15158.741974] LNet: Added LNI 192.168.201.127@tcp13 [8/256/0/180] [15165.573376] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15165.603470] LNet: Added LNI 192.168.201.127@tcp14 [8/256/0/180] [15171.589569] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15171.639992] LNet: Added LNI 192.168.201.127@tcp15 [8/256/0/180] [15179.125521] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15179.166813] LNet: Added LNI 192.168.201.127@tcp16 [8/256/0/180] [15187.641405] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15187.748178] LNet: Added LNI 192.168.201.127@tcp17 [8/256/0/180] [15194.756064] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15194.778720] LNet: Added LNI 192.168.201.127@tcp18 [8/256/0/180] [15200.049777] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15200.108895] LNet: Added LNI 192.168.201.127@tcp19 [8/256/0/180] [15206.899352] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15206.968294] LNet: Added LNI 192.168.201.127@tcp20 [8/256/0/180] [15213.997856] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15214.065677] LNet: Added LNI 192.168.201.127@tcp21 [8/256/0/180] [15220.821362] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15220.862022] LNet: Added LNI 192.168.201.127@tcp22 [8/256/0/180] [15229.009523] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15229.035593] LNet: Added LNI 192.168.201.127@tcp23 [8/256/0/180] [15236.044691] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15236.077507] LNet: Added LNI 192.168.201.127@tcp24 [8/256/0/180] [15242.416478] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15242.446821] LNet: Added LNI 192.168.201.127@tcp25 [8/256/0/180] [15249.886754] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15249.939520] LNet: Added LNI 192.168.201.127@tcp26 [8/256/0/180] [15257.526107] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15257.576538] LNet: Added LNI 192.168.201.127@tcp27 [8/256/0/180] [15265.930725] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15265.962274] LNet: Added LNI 192.168.201.127@tcp28 [8/256/0/180] [15273.863515] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15273.882321] LNet: Added LNI 192.168.201.127@tcp29 [8/256/0/180] [15283.065668] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15283.092580] LNet: Added LNI 192.168.201.127@tcp30 [8/256/0/180] [15292.051909] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15292.116276] LNet: Added LNI 192.168.201.127@tcp31 [8/256/0/180] [15300.586627] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15300.619659] LNet: Added LNI 192.168.201.127@tcp32 [8/256/0/180] [15309.481993] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15309.532983] LNet: Added LNI 192.168.201.127@tcp33 [8/256/0/180] [15318.790366] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15318.812483] LNet: Added LNI 192.168.201.127@tcp34 [8/256/0/180] [15327.275869] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15327.321522] LNet: Added LNI 192.168.201.127@tcp35 [8/256/0/180] [15335.414800] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15335.456742] LNet: Added LNI 192.168.201.127@tcp36 [8/256/0/180] [15344.379607] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15344.408937] LNet: Added LNI 192.168.201.127@tcp37 [8/256/0/180] [15352.095331] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15352.125864] LNet: Added LNI 192.168.201.127@tcp38 [8/256/0/180] [15360.961539] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15360.994634] LNet: Added LNI 192.168.201.127@tcp39 [8/256/0/180] [15369.239764] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15369.266121] LNet: Added LNI 192.168.201.127@tcp40 [8/256/0/180] [15377.341828] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15377.414744] LNet: Added LNI 192.168.201.127@tcp41 [8/256/0/180] [15386.079680] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15386.151504] LNet: Added LNI 192.168.201.127@tcp42 [8/256/0/180] [15395.361939] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15395.390981] LNet: Added LNI 192.168.201.127@tcp43 [8/256/0/180] [15404.759142] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15404.798817] LNet: Added LNI 192.168.201.127@tcp44 [8/256/0/180] [15414.599499] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15414.624642] LNet: Added LNI 192.168.201.127@tcp45 [8/256/0/180] [15423.611587] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15423.636942] LNet: Added LNI 192.168.201.127@tcp46 [8/256/0/180] [15433.125855] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15433.180398] LNet: Added LNI 192.168.201.127@tcp47 [8/256/0/180] [15442.091780] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15442.150861] LNet: Added LNI 192.168.201.127@tcp48 [8/256/0/180] [15451.914657] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15451.943744] LNet: Added LNI 192.168.201.127@tcp49 [8/256/0/180] [15465.097953] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15465.148840] LNet: Added LNI 192.168.201.127@tcp50 [8/256/0/180] [15489.002502] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15489.013751] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15489.033491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15494.642408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15494.658387] Lustre: Skipped 5 previous similar messages [15499.752523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15503.327198] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15503.476960] Lustre: server umount lustre-MDT0000 complete [15507.761229] LustreError: 473396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455347 with bad export cookie 8606778333053550452 [15507.770549] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15507.772372] LustreError: 473396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15508.092955] Lustre: server umount lustre-MDT0001 complete [15526.367183] Lustre: 472677:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788455350/real 1788455350] req@00000000613ef844 x1875330861299072/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788455366 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15526.423068] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15526.450369] Lustre: Skipped 2 previous similar messages [15529.121306] Lustre: 472678:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788455353/real 1788455353] req@000000002bed423f x1875330861299200/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788455369 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15531.388737] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [15531.487980] Lustre: 472680:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788455355/real 1788455355] req@00000000c63a3b1e x1875330861299328/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788455371 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15533.607354] Lustre: 472680:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788455357/real 1788455357] req@0000000063ad39c6 x1875330861299584/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788455373 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15538.659490] Lustre: 472677:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788455362/real 1788455362] req@00000000957699e8 x1875330861299712/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788455378 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15542.685204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15552.304873] LustreError: 475372:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15552.322458] LustreError: 475372:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [15552.347245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15556.388284] LustreError: 475373:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15558.125986] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [15560.484915] LustreError: 475372:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15560.499695] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15565.802590] LustreError: 475373:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15568.426380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15568.980907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15570.994215] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15574.992767] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [15584.547713] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15593.165165] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15622.116767] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15622.129193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15622.144649] Lustre: Skipped 1 previous similar message [15622.163262] Lustre: Skipped 3 previous similar messages [15626.720962] Lustre: server umount lustre-OST0000 complete [15632.353596] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15632.372708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15632.377280] Lustre: Skipped 1 previous similar message [15632.395517] Lustre: Skipped 1 previous similar message [15638.424470] Lustre: server umount lustre-MDT0000 complete [15642.598321] LustreError: 500047:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15642.656928] LustreError: 500047:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15643.895993] LustreError: 473396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455484 with bad export cookie 8606778333053553098 [15643.906785] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15643.917837] LustreError: 473396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15644.627847] Lustre: server umount lustre-MDT0001 complete [15661.980758] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [15665.756942] Key type lgssc unregistered [15666.492790] LNet: 502480:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15666.509061] LNetError: 502480:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15666.519878] LNet: Removed LNI 192.168.201.127@tcp [15666.525720] LNet: Removed LNI 192.168.201.127@tcp10 [15666.530552] LNet: Removed LNI 192.168.201.127@tcp11 [15666.537579] LNet: Removed LNI 192.168.201.127@tcp12 [15666.542434] LNet: Removed LNI 192.168.201.127@tcp13 [15666.543909] LNet: Removed LNI 192.168.201.127@tcp14 [15666.550477] LNet: Removed LNI 192.168.201.127@tcp15 [15666.557884] LNet: Removed LNI 192.168.201.127@tcp16 [15666.562080] LNet: Removed LNI 192.168.201.127@tcp17 [15666.565576] LNet: Removed LNI 192.168.201.127@tcp18 [15666.569432] LNet: Removed LNI 192.168.201.127@tcp19 [15666.573718] LNet: Removed LNI 192.168.201.127@tcp20 [15666.576528] LNet: Removed LNI 192.168.201.127@tcp21 [15666.587076] LNet: Removed LNI 192.168.201.127@tcp22 [15666.597415] LNet: Removed LNI 192.168.201.127@tcp23 [15666.600513] LNet: Removed LNI 192.168.201.127@tcp24 [15666.603801] LNet: Removed LNI 192.168.201.127@tcp25 [15666.607568] LNet: Removed LNI 192.168.201.127@tcp26 [15666.610598] LNet: Removed LNI 192.168.201.127@tcp27 [15666.615798] LNet: Removed LNI 192.168.201.127@tcp28 [15666.628048] LNet: Removed LNI 192.168.201.127@tcp29 [15666.633770] LNet: Removed LNI 192.168.201.127@tcp30 [15666.641621] LNet: Removed LNI 192.168.201.127@tcp31 [15666.655591] LNet: Removed LNI 192.168.201.127@tcp32 [15666.661887] LNet: Removed LNI 192.168.201.127@tcp33 [15666.672612] LNet: Removed LNI 192.168.201.127@tcp34 [15666.686982] LNet: Removed LNI 192.168.201.127@tcp35 [15666.691994] LNet: Removed LNI 192.168.201.127@tcp36 [15666.694688] LNet: Removed LNI 192.168.201.127@tcp37 [15666.703530] LNet: Removed LNI 192.168.201.127@tcp38 [15666.708993] LNet: Removed LNI 192.168.201.127@tcp39 [15666.713545] LNet: Removed LNI 192.168.201.127@tcp40 [15666.716648] LNet: Removed LNI 192.168.201.127@tcp41 [15666.721451] LNet: Removed LNI 192.168.201.127@tcp42 [15666.728805] LNet: Removed LNI 192.168.201.127@tcp43 [15666.737547] LNet: Removed LNI 192.168.201.127@tcp44 [15666.752145] LNet: Removed LNI 192.168.201.127@tcp45 [15666.753797] LNet: Removed LNI 192.168.201.127@tcp46 [15666.764133] LNet: Removed LNI 192.168.201.127@tcp47 [15666.772913] LNet: Removed LNI 192.168.201.127@tcp48 [15666.785951] LNet: Removed LNI 192.168.201.127@tcp49 [15666.813164] LNet: Removed LNI 192.168.201.127@tcp50 [15667.984624] Key type .llcrypt unregistered [15667.986076] Key type ._llcrypt unregistered [15691.197923] Key type ._llcrypt registered [15691.199715] Key type .llcrypt registered [15691.398370] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [15713.120374] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [15715.127573] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15715.670188] alg: No test for adler32 (adler32-zlib) [15716.912768] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [15717.325041] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [15719.023272] Key type lgssc registered [15719.986921] Lustre: Echo OBD driver; http://www.lustre.org/ [15765.558425] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15765.600739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15766.860945] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15766.925915] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15767.042557] Lustre: lustre-MDT0000: new disk, initializing [15767.232468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15767.286240] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15771.606460] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [15785.302879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15785.508527] Lustre: 505676:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15785.676409] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15785.705073] Lustre: Skipped 1 previous similar message [15786.016972] Lustre: lustre-MDT0001: new disk, initializing [15786.208699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15786.322995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15786.362430] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15792.989955] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [15805.509190] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15812.494624] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15820.509726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15820.687471] Lustre: *** cfs_fail_loc=90f, val=0*** [15820.761611] Lustre: lustre-OST0000: new disk, initializing [15820.768389] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15820.777725] Lustre: 507692:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15820.902035] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15827.474284] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [15830.687987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15830.704904] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15830.911212] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15838.087354] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15846.380285] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15846.389623] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15846.393204] LustreError: Skipped 1 previous similar message [15846.409941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15846.418767] Lustre: Skipped 1 previous similar message [15846.446535] Lustre: Skipped 1 previous similar message [15851.491036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15851.504397] Lustre: Skipped 1 previous similar message [15852.391779] Lustre: server umount lustre-OST0000 complete [15856.609544] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15856.636537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15856.650697] Lustre: Skipped 1 previous similar message [15861.732662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15861.744169] Lustre: Skipped 1 previous similar message [15862.684820] Lustre: server umount lustre-MDT0000 complete [15866.857667] LustreError: 505666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455707 with bad export cookie 5965824388388800368 [15866.865356] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15866.892142] LustreError: 505666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15867.276330] Lustre: server umount lustre-MDT0001 complete [15879.641479] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [15882.950368] Key type lgssc unregistered [15883.288868] LNet: 509599:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15883.299042] LNetError: 509599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15883.314175] LNet: Removed LNI 192.168.201.127@tcp [15884.487185] Key type .llcrypt unregistered [15884.491829] Key type ._llcrypt unregistered [15912.802981] Key type ._llcrypt registered [15912.805399] Key type .llcrypt registered [15912.905617] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [15913.985414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15914.047233] alg: No test for adler32 (adler32-zlib) [15915.225230] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [15915.505623] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [15917.199666] Key type lgssc registered [15919.077585] Lustre: Echo OBD driver; http://www.lustre.org/ [15927.281871] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15927.322152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15928.720640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15933.886438] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [15938.038684] LustreError: 510870:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15938.058473] LustreError: 510870:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [15943.144550] LustreError: 510869:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15945.666932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15946.132649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15951.970633] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [15961.107809] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15969.009641] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15976.167520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15976.533165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15984.207085] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [15992.739633] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16007.146643] loop0: detected capacity change from 0 to 204800000 [16007.386091] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16007.410842] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16007.422827] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16014.531594] loop0: detected capacity change from 0 to 204800000 [16014.738925] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16014.752488] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16014.771473] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16020.623173] loop0: detected capacity change from 0 to 204800000 [16026.681298] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16026.851400] Lustre: fs15246-OST0000: new disk, initializing [16026.855370] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16026.871723] Lustre: 514467:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16026.975909] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16034.453500] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [16045.250358] loop1: detected capacity change from 0 to 204800000 [16050.988280] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16051.094952] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16051.128226] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16051.191691] Lustre: fs15246-MDT0000: new disk, initializing [16051.351312] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16051.392209] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16051.528472] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16052.576503] Lustre: 515588:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000005bdb42b6 x1875331781806720/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788455908 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16056.045560] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [16061.683402] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16061.688899] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16061.715755] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16080.884118] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16080.914111] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16086.014200] LustreError: 510865:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16086.176423] Lustre: server umount fs15246-MDT0000 complete [16091.107593] LustreError: 510866:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16096.252558] LustreError: 510864:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16097.828700] Lustre: server umount fs15246-OST0000 complete [16114.147238] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16114.158858] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16114.174684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16116.198514] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16116.210135] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16116.226105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16119.462422] Lustre: server umount lustre-OST0000 complete [16125.409412] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16125.426625] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16125.462081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16125.476773] Lustre: Skipped 1 previous similar message [16130.032531] Lustre: server umount lustre-MDT0000 complete [16132.067785] LustreError: 510870:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16132.081260] LustreError: 510870:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [16134.849229] LustreError: 510849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788455975 with bad export cookie 16244654109554347527 [16134.859215] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16134.868772] LustreError: 510849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16135.372738] Lustre: server umount lustre-MDT0001 complete [16149.578485] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [16153.934283] Key type lgssc unregistered [16154.302082] LNet: 518686:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16154.320971] LNetError: 518686:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16154.350942] LNet: Removed LNI 192.168.201.127@tcp [16155.593094] Key type .llcrypt unregistered [16155.605096] Key type ._llcrypt unregistered [16179.488268] Key type ._llcrypt registered [16179.490398] Key type .llcrypt registered [16179.577071] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [16181.254106] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16181.314968] alg: No test for adler32 (adler32-zlib) [16182.408865] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [16182.726859] LNet: Added LNI 192.168.201.127@tcp [8/256/0/180] [16184.431431] Key type lgssc registered [16185.761444] Lustre: Echo OBD driver; http://www.lustre.org/ [16234.233703] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [16245.617129] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [16285.207742] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16285.235684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16286.471518] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16286.507083] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16286.572569] Lustre: lustre-MDT0000: new disk, initializing [16286.645965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16286.664596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16290.618747] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [16303.292768] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16303.421988] Lustre: 523156:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16303.476911] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16303.481478] Lustre: Skipped 1 previous similar message [16303.599758] Lustre: lustre-MDT0001: new disk, initializing [16303.768843] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16303.811890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16303.823799] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16309.179768] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [16320.466737] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16330.665542] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16343.190522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16343.721387] Lustre: lustre-OST0000: new disk, initializing [16343.753929] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16343.770611] Lustre: 525124:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16343.946983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16352.569199] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_default_debug -1 all [16353.943582] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16353.955868] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16354.133147] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16362.934909] Lustre: DEBUG MARKER: oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16367.854676] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16368.182215] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16373.950839] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16374.152359] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16379.872849] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16379.873168] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16379.882271] LustreError: Skipped 1 previous similar message [16379.894820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16379.924964] Lustre: Skipped 1 previous similar message [16379.949165] Lustre: Skipped 1 previous similar message [16382.680058] Lustre: server umount lustre-OST0000 complete [16390.113403] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16390.117102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16390.128785] Lustre: Skipped 1 previous similar message [16395.233060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16395.238671] Lustre: Skipped 2 previous similar messages [16400.352393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16400.370066] Lustre: Skipped 1 previous similar message [16401.377339] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16401.847303] Lustre: server umount lustre-MDT0000 complete [16406.678816] LustreError: 525123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788456246 with bad export cookie 1670583332863803416 [16406.684670] LustreError: MGC192.168.201.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16406.688560] LustreError: 525123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16407.371569] Lustre: server umount lustre-MDT0001 complete [16436.337322] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing set_hostid [16448.380257] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing load_modules_local [16489.471307] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16185 sec ========= 13:25:28 (1788456328) [16491.426976] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 13:25:30 (1788456330) === [16495.332538] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 13:25:34 (1788456334) === [16544.235474] Lustre: DEBUG MARKER: oleg127-server.virtnet: executing unload_modules_local [16549.358168] Key type lgssc unregistered [16549.810230] LNet: 531775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16549.815908] LNetError: 531775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16549.836329] LNet: Removed LNI 192.168.201.127@tcp [16551.335972] Key type .llcrypt unregistered [16551.338991] Key type ._llcrypt unregistered