[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 497181903 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002193] x2apic enabled [ 0.003004] Switched APIC routing to physical x2apic. [ 0.004009] kvm-guest: setup PV IPIs [ 0.005993] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.006000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.006022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.007011] pid_max: default: 32768 minimum: 301 [ 0.008106] LSM: Security Framework initializing [ 0.009057] Yama: becoming mindful. [ 0.011045] SELinux: Initializing. [ 0.012081] *** VALIDATE selinux *** [ 0.019637] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023493] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024154] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.025118] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026117] *** VALIDATE tmpfs *** [ 0.028250] *** VALIDATE proc *** [ 0.029209] *** VALIDATE cgroup *** [ 0.029852] *** VALIDATE cgroup2 *** [ 0.030285] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.031111] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.032005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.033027] Spectre V2 : User space: Vulnerable [ 0.034008] Speculative Store Bypass: Vulnerable [ 0.036725] debug: unmapping init [mem 0xffffffff9e059000-0xffffffff9e060fff] [ 0.038169] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.039679] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.040022] ... version: 2 [ 0.041011] ... bit width: 48 [ 0.041987] ... generic registers: 4 [ 0.042011] ... value mask: 0000ffffffffffff [ 0.043012] ... max period: 00007fffffffffff [ 0.044024] ... fixed-purpose events: 3 [ 0.044811] ... event mask: 000000070000000f [ 0.045305] rcu: Hierarchical SRCU implementation. [ 0.047429] smp: Bringing up secondary CPUs ... [ 0.048377] x86: Booting SMP configuration: [ 0.049016] .... node #0, CPUs: #1 #2 #3 [ 0.052189] smp: Brought up 1 node, 4 CPUs [ 0.054014] smpboot: Max logical packages: 1 [ 0.055016] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.125360] node 0 deferred pages initialised in 67ms [ 0.127108] devtmpfs: initialized [ 0.128039] x86/mm: Memory block size: 128MB [ 0.129966] gcov: version magic: 0x41383552 [ 0.130562] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.131106] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.132253] pinctrl core: initialized pinctrl subsystem [ 0.133241] [ 0.133803] ************************************************************* [ 0.134016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.135014] ** ** [ 0.136014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.137015] ** ** [ 0.138014] ** This means that this kernel is built to expose internal ** [ 0.139010] ** IOMMU data structures, which may compromise security on ** [ 0.140014] ** your system. ** [ 0.141013] ** ** [ 0.142011] ** If you see this message and you are not debugging the ** [ 0.143008] ** kernel, report this immediately to your vendor! ** [ 0.144011] ** ** [ 0.145011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.146010] ************************************************************* [ 0.147634] NET: Registered protocol family 16 [ 0.148410] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.149053] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.150065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.151415] cpuidle: using governor menu [ 0.153591] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.155449] PCI: Using configuration type 1 for base access [ 0.157126] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.166103] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.167025] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.169033] cryptd: max_cpu_qlen set to 1000 [ 0.172315] ACPI: Added _OSI(Module Device) [ 0.173010] ACPI: Added _OSI(Processor Device) [ 0.173933] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.174013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.178730] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.181045] ACPI: Interpreter enabled [ 0.182066] ACPI: PM: (supports S0 S3 S4 S5) [ 0.183049] ACPI: Using IOAPIC for interrupt routing [ 0.184175] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.187372] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.196349] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.198044] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.200014] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.203087] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.207089] acpiphp: Slot [2] registered [ 0.208139] acpiphp: Slot [5] registered [ 0.210106] acpiphp: Slot [6] registered [ 0.211095] acpiphp: Slot [7] registered [ 0.212071] acpiphp: Slot [8] registered [ 0.213014] acpiphp: Slot [9] registered [ 0.213965] acpiphp: Slot [10] registered [ 0.214081] acpiphp: Slot [3] registered [ 0.215000] acpiphp: Slot [4] registered [ 0.215000] acpiphp: Slot [11] registered [ 0.215000] acpiphp: Slot [12] registered [ 0.215000] acpiphp: Slot [13] registered [ 0.215000] acpiphp: Slot [14] registered [ 0.215000] acpiphp: Slot [15] registered [ 0.215000] acpiphp: Slot [16] registered [ 0.215000] acpiphp: Slot [17] registered [ 0.215000] acpiphp: Slot [18] registered [ 0.215000] acpiphp: Slot [19] registered [ 0.215000] acpiphp: Slot [20] registered [ 0.215000] acpiphp: Slot [21] registered [ 0.215000] acpiphp: Slot [22] registered [ 0.215000] acpiphp: Slot [23] registered [ 0.215000] acpiphp: Slot [24] registered [ 0.215000] acpiphp: Slot [25] registered [ 0.215000] acpiphp: Slot [26] registered [ 0.215000] acpiphp: Slot [27] registered [ 0.215000] acpiphp: Slot [28] registered [ 0.215000] acpiphp: Slot [29] registered [ 0.215000] acpiphp: Slot [30] registered [ 0.220106] acpiphp: Slot [31] registered [ 0.222069] PCI host bridge to bus 0000:00 [ 0.224020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.226021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.228027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.231028] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.233014] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.235017] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.236190] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.238794] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.241945] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.249016] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.255073] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.257019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.259015] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.261018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.263540] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.266808] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.269044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.271676] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.277014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.289019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.294013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.298969] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.308018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.314017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.330017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.339695] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.344013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.347012] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.366019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.377178] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.382012] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.394033] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.417017] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.427460] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.437019] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.441012] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.464017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.473209] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.480027] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.492017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.515024] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.532084] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.538015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.543242] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.561026] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.575162] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.577377] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.580377] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.583413] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.585255] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.589215] iommu: Default domain type: Passthrough [ 0.591450] SCSI subsystem initialized [ 0.592198] ACPI: bus type USB registered [ 0.594125] usbcore: registered new interface driver usbfs [ 0.596088] usbcore: registered new interface driver hub [ 0.598100] usbcore: registered new device driver usb [ 0.600184] pps_core: LinuxPPS API ver. 1 registered [ 0.601011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.605068] PTP clock support registered [ 0.607107] EDAC MC: Ver: 3.0.0 [ 0.609134] PCI: Using ACPI for IRQ routing [ 0.610998] NetLabel: Initializing [ 0.613013] NetLabel: domain hash size = 128 [ 0.614009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.616098] NetLabel: unlabeled traffic allowed by default [ 0.618175] vgaarb: loaded [ 0.620343] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.621074] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.628070] clocksource: Switched to clocksource kvm-clock [ 0.729750] VFS: Disk quotas dquot_6.6.0 [ 0.731402] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.733819] *** VALIDATE ramfs *** [ 0.734739] *** VALIDATE hugetlbfs *** [ 0.736346] pnp: PnP ACPI init [ 0.738794] pnp: PnP ACPI: found 6 devices [ 0.759401] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.762868] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.765093] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.767465] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.769917] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.772361] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.775270] NET: Registered protocol family 2 [ 0.777715] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.782629] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.785330] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.790095] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.794606] TCP: Hash tables configured (established 65536 bind 65536) [ 0.797884] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.801050] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.803887] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.806810] NET: Registered protocol family 1 [ 0.809963] RPC: Registered named UNIX socket transport module. [ 0.811596] RPC: Registered udp transport module. [ 0.813010] RPC: Registered tcp transport module. [ 0.814210] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.816185] NET: Registered protocol family 44 [ 0.817843] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.820158] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.822513] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.824969] PCI: CLS 0 bytes, default 64 [ 0.826702] Unpacking initramfs... [ 2.168118] debug: unmapping init [mem 0xffff903e3cc54000-0xffff903e3ffbffff] [ 2.171513] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.173852] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.176668] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.695503] Initialise system trusted keyrings [ 2.697053] Key type blacklist registered [ 2.698897] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.707842] zbud: loaded [ 2.710762] *** VALIDATE nfs *** [ 2.711956] *** VALIDATE nfs4 *** [ 2.713518] pstore: using deflate compression [ 2.717320] Platform Keyring initialized [ 2.814781] NET: Registered protocol family 38 [ 2.816670] Key type asymmetric registered [ 2.818190] Asymmetric key parser 'x509' registered [ 2.819995] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.822877] io scheduler mq-deadline registered [ 2.824525] io scheduler kyber registered [ 2.826321] io scheduler bfq registered [ 2.828220] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.831415] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.834206] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.836917] ACPI: Power Button [PWRF] [ 2.842184] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.848028] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.859094] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.865715] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.878988] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.907223] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.935581] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.940691] Non-volatile memory driver v1.3 [ 2.942191] Linux agpgart interface v0.103 [ 2.971563] virtio_blk virtio1: [vda] 149760 512-byte logical blocks (76.7 MB/73.1 MiB) [ 2.974272] vda: detected capacity change from 0 to 76677120 [ 2.987762] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.989506] vdb: detected capacity change from 0 to 1073741824 [ 3.005370] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.007094] vdc: detected capacity change from 0 to 2621440000 [ 3.020038] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.022599] vdd: detected capacity change from 0 to 2621440000 [ 3.039241] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.041916] vde: detected capacity change from 0 to 4294967296 [ 3.054414] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.056256] vdf: detected capacity change from 0 to 4294967296 [ 3.062751] libphy: Fixed MDIO Bus: probed [ 3.075176] usbcore: registered new interface driver usbserial_generic [ 3.077598] usbserial: USB Serial support registered for generic [ 3.079922] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.084592] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.086592] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.090326] mousedev: PS/2 mouse device common for all mice [ 3.092633] rtc_cmos 00:05: RTC can wake from S4 [ 3.094970] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.098238] rtc_cmos 00:05: registered as rtc0 [ 3.099860] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.103583] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.105686] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.106823] intel_pstate: CPU model not supported [ 3.111473] hid: raw HID events driver (C) Jiri Kosina [ 3.113051] usbcore: registered new interface driver usbhid [ 3.114517] usbhid: USB HID core driver [ 3.115524] drop_monitor: Initializing network drop monitor service [ 3.117458] Initializing XFRM netlink socket [ 3.118776] NET: Registered protocol family 10 [ 3.121380] Segment Routing with IPv6 [ 3.122782] NET: Registered protocol family 17 [ 3.124755] mpls_gso: MPLS GSO support [ 3.130503] RAS: Correctable Errors collector initialized. [ 3.132716] AVX version of gcm_enc/dec engaged. [ 3.134387] AES CTR mode by8 optimization enabled [ 3.210121] sched_clock: Marking stable (3210052849, 0)->(4082816099, -872763250) [ 3.213743] registered taskstats version 1 [ 3.215766] Loading compiled-in X.509 certificates [ 3.217644] zswap: loaded using pool lzo/zbud [ 3.241167] Key type big_key registered [ 3.252868] Key type encrypted registered [ 3.254751] ima: No TPM chip found, activating TPM-bypass! [ 3.257065] ima: Allocated hash algorithm: sha1 [ 3.258848] ima: No architecture policies found [ 3.260675] evm: Initialising EVM extended attributes: [ 3.262654] evm: security.selinux [ 3.263921] evm: security.ima [ 3.265099] evm: security.capability [ 3.266489] evm: HMAC attrs: 0x1 [ 3.268961] rtc_cmos 00:05: setting system clock to 2026-09-03 12:02:30 UTC (1788436950) [ 3.275216] debug: unmapping init [mem 0xffffffff9f003000-0xffffffff9f1fffff] [ 3.278669] debug: unmapping init [mem 0xffffffff9dd82000-0xffffffff9e058fff] [ 3.287111] Write protecting the kernel read-only data: 28672k [ 3.290484] debug: unmapping init [mem 0xffffffff9c403000-0xffffffff9c5fffff] [ 3.293433] debug: unmapping init [mem 0xffffffff9cd14000-0xffffffff9cdfffff] [ 3.327990] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.337405] systemd[1]: Detected virtualization kvm. [ 3.339308] systemd[1]: Detected architecture x86-64. [ 3.341477] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.369397] systemd[1]: No hostname configured. [ 3.371055] systemd[1]: Set hostname to . [ 3.373061] random: systemd: uninitialized urandom read (16 bytes read) [ 3.375528] systemd[1]: Initializing machine ID from random generator. [ 3.506536] random: systemd: uninitialized urandom read (16 bytes read) [ 3.509671] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.513989] random: systemd: uninitialized urandom read (16 bytes read) [ 3.520596] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ 3.525327] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Sockets. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.012589] device-mapper: uevent: version 1.0.3 [ 4.014399] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.642385] random: fast init done [ 4.679303] virtio_net virtio0 ens2: renamed from eth0 [ 4.698115] scsi host0: ata_piix [ 4.761416] scsi host1: ata_piix [ 4.763154] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.765651] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.335834] dracut-initqueue[583]: RTNETLINK answers: File exists [ 9.780816] random: crng init done [ 9.782332] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.083913] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.240535] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.443699] SELinux: Disabled at runtime. [ 11.499383] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.506532] systemd[1]: Detected virtualization kvm. [ 11.508092] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.918890] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.921278] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.925537] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.929234] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.933222] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 11.940119] systemd[1]: Starting Journal Service... Starting Journal Service... [ 11.944157] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-sshd\x2dkeygen.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... Starting Apply Kernel Variables... [ 12.107031] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.357602] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.663378] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.717211] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.750175] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.764500] EDAC sbridge: Ver: 1.1.2 [ 14.359690] Key type dns_resolver registered [ 14.656317] NFS: Registering the id_resolver key type [ 14.658040] Key type id_resolver registered [ 14.659644] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg133-server login: [ 44.621542] hrtimer: interrupt took 12530070 ns [ 66.555780] libcfs: loading out-of-tree module taints kernel. [ 66.676964] Key type ._llcrypt registered [ 66.678437] Key type .llcrypt registered [ 66.823691] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 87.332302] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 89.098559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 89.114255] alg: No test for adler32 (adler32-zlib) [ 90.693723] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 91.729459] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 93.487175] Key type lgssc registered [ 95.269016] Lustre: Echo OBD driver; http://www.lustre.org/ [ 114.636401] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 156.550812] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 171.445968] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 171.497268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 172.730232] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 172.751045] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 172.804997] Lustre: lustre-MDT0000: new disk, initializing [ 172.870134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 172.887830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 177.664440] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 193.185446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 193.286325] Lustre: 6512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 193.307852] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 193.311737] Lustre: Skipped 1 previous similar message [ 193.405650] Lustre: lustre-MDT0001: new disk, initializing [ 193.456598] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 193.487613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 193.507275] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 198.497445] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 204.253988] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 215.245630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 215.521334] Lustre: lustre-OST0000: new disk, initializing [ 215.523913] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 215.530104] Lustre: 8451:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 215.597290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 220.200548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 220.208974] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 220.286782] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 222.162980] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 237.878454] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 238.070481] Lustre: lustre-OST0001: new disk, initializing [ 238.081617] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 238.088599] Lustre: 9523:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 238.197038] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 243.745848] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 243.752858] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 243.852705] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 245.754286] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 260.931197] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 267.896317] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 274.913666] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing check_logdir /tmp/testlogs/ [ 280.559331] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing yml_node [ 285.269230] Lustre: DEBUG MARKER: Client: 2.17.58.2 [ 288.509970] Lustre: DEBUG MARKER: MDS: 2.17.58.2 [ 291.290598] Lustre: DEBUG MARKER: OSS: 2.17.58.2 [ 293.259297] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 3 08:07:18 EDT 2026 [ 311.711945] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 313.606255] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 330.738870] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 330.751645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 330.760370] Lustre: Skipped 3 previous similar messages [ 330.778710] Lustre: Skipped 3 previous similar messages [ 335.842142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 335.869243] Lustre: Skipped 3 previous similar messages [ 336.022286] Lustre: server umount lustre-MDT0000 complete [ 340.079753] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788437287 with bad export cookie 9245748589876812136 [ 340.086473] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 340.090665] LustreError: 6506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 340.962886] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 340.976711] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 340.991686] Lustre: Skipped 1 previous similar message [ 346.081886] LustreError: 6520:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 346.083807] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 346.103838] LustreError: 6520:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 346.116688] Lustre: Skipped 1 previous similar message [ 346.633601] Lustre: server umount lustre-MDT0001 complete [ 350.925567] Lustre: server umount lustre-OST0000 complete [ 355.061271] Lustre: server umount lustre-OST0001 complete [ 363.219070] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 373.366173] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 413.450417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 413.717535] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 413.775283] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 413.851202] Lustre: lustre-MDT0000: new disk, initializing [ 413.936322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 413.955924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 419.197251] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 430.517427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 430.763768] Lustre: 15079:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 430.867801] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 430.871143] Lustre: Skipped 1 previous similar message [ 430.971945] Lustre: lustre-MDT0001: new disk, initializing [ 431.108880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 431.195492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 431.199992] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 436.643652] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 447.675741] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 454.408603] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 462.196873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 462.573379] Lustre: lustre-OST0000: new disk, initializing [ 462.577383] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 462.584186] Lustre: 17050:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 462.720750] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 464.643242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 464.657375] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 464.811750] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 469.787577] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 480.396594] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 485.664856] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 485.843372] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 490.713857] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 490.908856] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 495.593542] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 495.601175] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 495.628301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 495.639123] Lustre: Skipped 2 previous similar messages [ 505.828316] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 505.834396] Lustre: Skipped 2 previous similar messages [ 507.359268] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 507.638781] Lustre: server umount lustre-OST0000 complete [ 512.995661] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 513.005580] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 513.037687] Lustre: Skipped 1 previous similar message [ 518.668795] Lustre: server umount lustre-MDT0000 complete [ 521.184921] LustreError: 15092:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 521.205728] LustreError: 15092:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 522.545080] LustreError: 15073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788437469 with bad export cookie 9245748589876813669 [ 522.556319] LustreError: 15073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 522.562457] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 522.938754] Lustre: server umount lustre-MDT0001 complete [ 527.488675] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 08:11:13 (1788437473) [ 530.043559] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 532.115151] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 08:11:17 (1788437477) [ 539.357858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 539.954498] LustreError: 19462:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 540.056794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 545.255373] LustreError: 19463:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 545.273993] LustreError: 19463:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 545.586500] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 550.371543] LustreError: 19462:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 554.922795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 560.328760] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 568.880537] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 575.317451] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 582.543247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 589.560605] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 597.879948] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 611.812203] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 611.816128] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 611.825578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 611.835926] Lustre: Skipped 2 previous similar messages [ 611.848682] Lustre: Skipped 5 previous similar messages [ 615.026631] Lustre: server umount lustre-OST0000 complete [ 622.050658] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 622.057642] Lustre: Skipped 1 previous similar message [ 622.062367] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 622.077290] LustreError: Skipped 1 previous similar message [ 624.693154] Lustre: server umount lustre-MDT0000 complete [ 627.175158] LustreError: 19462:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 627.207896] LustreError: 19462:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 628.535738] LustreError: 19443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788437575 with bad export cookie 9245748589876814551 [ 628.559548] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 628.889642] Lustre: server umount lustre-MDT0001 complete [ 638.807779] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 641.328370] Key type lgssc unregistered [ 641.606917] LNet: 22997:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 641.612576] LNetError: 22997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 641.624818] LNet: Removed LNI 192.168.201.133@tcp [ 642.395229] Key type .llcrypt unregistered [ 642.398229] Key type ._llcrypt unregistered [ 670.880566] Key type ._llcrypt registered [ 670.884239] Key type .llcrypt registered [ 671.027774] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 672.129503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 672.322851] alg: No test for adler32 (adler32-zlib) [ 673.456817] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 673.805241] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 675.495342] Key type lgssc registered [ 676.994867] Lustre: Echo OBD driver; http://www.lustre.org/ [ 685.240583] loop: module loaded [ 685.323218] loop0: detected capacity change from 0 to 204800000 [ 685.604048] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 685.640153] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 686.066757] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 692.676416] loop0: detected capacity change from 0 to 204800000 [ 692.952233] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 692.971310] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 692.999629] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 699.305959] loop0: detected capacity change from 0 to 204800000 [ 705.366859] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 705.393991] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 706.757694] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 706.827652] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 706.975300] Lustre: lustre-MDT0000: new disk, initializing [ 707.116619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 707.138822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 712.571954] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 721.548718] loop1: detected capacity change from 0 to 204800000 [ 727.375769] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 727.729831] Lustre: lustre-OST07c6: new disk, initializing [ 727.732722] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 727.737370] Lustre: Skipped 1 previous similar message [ 727.749464] Lustre: 26179:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 727.857592] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 734.384753] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 736.404314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 736.411563] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 736.455394] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 740.107369] Lustre: 25167:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 740.113584] Lustre: 25167:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 740.118494] Lustre: 25167:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 740.123434] Lustre: 25167:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 740.128622] Lustre: 25167:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 740.133550] Lustre: 25167:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 755.332647] Lustre: 25167:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 755.339664] Lustre: 25167:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 755.345446] Lustre: 25167:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 755.350559] Lustre: 25167:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 755.356374] Lustre: 25167:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 755.360990] Lustre: 25167:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 760.392729] Lustre: 25167:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 760.402872] Lustre: 25167:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 760.408291] Lustre: 25167:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 760.416503] Lustre: 25167:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 760.421928] Lustre: 25167:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 760.428687] Lustre: 25167:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 760.435616] Lustre: 25167:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 760.449636] Lustre: 25167:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 760.455764] Lustre: 25167:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 760.471744] Lustre: 25167:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 760.485272] Lustre: 25167:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 760.496551] Lustre: 25167:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 766.172498] Lustre: 25166:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 766.190635] Lustre: 25166:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 766.202347] Lustre: 25166:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 766.209320] Lustre: 25166:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 766.219571] Lustre: 25166:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 766.226552] Lustre: 25166:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 766.235627] Lustre: 25166:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 766.240397] Lustre: 25166:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 766.247297] Lustre: 25166:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 766.254666] Lustre: 25166:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 766.260169] Lustre: 25166:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 766.268133] Lustre: 25166:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 772.665368] Lustre: 25165:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 772.675370] Lustre: 25165:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 772.681416] Lustre: 25165:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 772.687149] Lustre: 25165:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 772.693305] Lustre: 25165:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 772.698058] Lustre: 25165:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 772.703043] Lustre: 25165:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 772.710056] Lustre: 25165:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 772.718282] Lustre: 25165:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 772.728191] Lustre: 25165:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 772.734750] Lustre: 25165:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 772.740124] Lustre: 25165:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 783.694552] Lustre: 26625:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 783.702727] Lustre: 26625:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 783.709101] Lustre: 26625:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 783.714386] Lustre: 26625:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 783.721789] Lustre: 26625:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 783.727025] Lustre: 26625:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 783.732263] Lustre: 26625:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 783.737040] Lustre: 26625:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 783.742444] Lustre: 26625:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 783.747345] Lustre: 26625:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 783.751154] Lustre: 26625:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 783.755772] Lustre: 26625:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 828.384175] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 828.385334] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 828.428644] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 833.509760] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 834.514951] Lustre: server umount lustre-OST07c6 complete [ 845.576412] Lustre: server umount lustre-MDT0000 complete [ 865.826753] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 869.390921] Key type lgssc unregistered [ 869.779861] LNet: 28734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 869.793670] LNetError: 28734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 869.823746] LNet: Removed LNI 192.168.201.133@tcp [ 870.962619] Key type .llcrypt unregistered [ 870.966336] Key type ._llcrypt unregistered [ 891.716178] Key type ._llcrypt registered [ 891.721892] Key type .llcrypt registered [ 891.866605] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 893.425218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 893.485052] alg: No test for adler32 (adler32-zlib) [ 894.576240] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 894.812330] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 896.522067] Key type lgssc registered [ 898.146707] Lustre: Echo OBD driver; http://www.lustre.org/ [ 906.740812] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 906.761042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 908.263365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 913.423685] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 918.502059] LustreError: 30004:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 918.516773] LustreError: 30004:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 922.713854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 923.164872] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 928.492520] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 937.039877] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 944.028157] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 951.714194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 952.073857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 953.146273] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 958.494947] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 966.913690] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 984.042899] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 984.045172] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 984.052351] LustreError: Skipped 1 previous similar message [ 984.055556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 984.084952] Lustre: Skipped 1 previous similar message [ 987.646677] Lustre: server umount lustre-OST0000 complete [ 994.272410] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 994.275349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 994.293275] Lustre: Skipped 1 previous similar message [ 994.305776] Lustre: Skipped 2 previous similar messages [ 999.320695] Lustre: server umount lustre-MDT0000 complete [ 999.395989] LustreError: 30004:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 999.418877] LustreError: 30004:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1003.674834] LustreError: 29983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788437950 with bad export cookie 11277222564127148244 [ 1003.686539] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1003.692058] LustreError: 29983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1004.219067] Lustre: server umount lustre-MDT0001 complete [ 1015.155633] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 1018.302486] Key type lgssc unregistered [ 1018.655924] LNet: 33739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1018.676803] LNetError: 33739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1019.753407] LNet: Removed LNI 192.168.201.133@tcp [ 1020.720534] Key type .llcrypt unregistered [ 1020.727829] Key type ._llcrypt unregistered [ 1039.153213] Key type ._llcrypt registered [ 1039.156269] Key type .llcrypt registered [ 1039.292648] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 1040.250901] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1040.274740] alg: No test for adler32 (adler32-zlib) [ 1041.311502] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1041.581171] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 1043.271162] Key type lgssc registered [ 1044.126471] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1050.773353] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1050.796333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1052.526307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1057.725802] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1061.865734] LustreError: 35005:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1061.897486] LustreError: 35005:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1066.977443] LustreError: 35006:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1067.031299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1067.358547] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1073.097187] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1081.498275] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1087.421648] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1094.431930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1094.697570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1097.785261] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1100.726785] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1109.088863] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1127.416512] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1127.420498] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1127.420933] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1127.436987] Lustre: Skipped 1 previous similar message [ 1130.344361] Lustre: server umount lustre-OST0000 complete [ 1137.633908] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1137.646069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1137.656300] Lustre: Skipped 1 previous similar message [ 1137.680575] Lustre: Skipped 2 previous similar messages [ 1140.762375] Lustre: server umount lustre-MDT0000 complete [ 1142.763933] LustreError: 37449:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1142.774796] LustreError: 37449:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1144.132278] LustreError: 34986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438091 with bad export cookie 3830058196700395858 [ 1144.134033] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1144.140293] LustreError: 34986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1144.545784] Lustre: server umount lustre-MDT0001 complete [ 1156.174658] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 1158.875433] Key type lgssc unregistered [ 1159.131695] LNet: 38562:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1159.139497] LNetError: 38562:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1159.155032] LNet: Removed LNI 192.168.201.133@tcp [ 1160.152738] Key type .llcrypt unregistered [ 1160.154785] Key type ._llcrypt unregistered [ 1181.408265] Key type ._llcrypt registered [ 1181.416169] Key type .llcrypt registered [ 1181.527806] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 1182.227907] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1182.312150] alg: No test for adler32 (adler32-zlib) [ 1183.325887] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1183.543976] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 1185.183219] Key type lgssc registered [ 1186.149483] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1192.779971] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1192.791897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1194.099311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1199.298732] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1204.712423] LustreError: 39953:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1204.734945] LustreError: 39953:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1207.776617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1208.207036] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1213.100630] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1221.090752] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1227.113389] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1233.544301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1233.809808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1238.976921] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1239.757402] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1246.387970] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1258.475331] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1258.489454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1259.357347] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.33@tcp (stopping) [ 1259.361167] Lustre: Skipped 2 previous similar messages [ 1263.586193] LustreError: 39954:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1263.627150] LustreError: 39954:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1265.375514] Lustre: server umount lustre-MDT0000 complete [ 1268.709722] LustreError: 40956:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1268.735847] LustreError: 40956:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1269.543912] LustreError: 39934:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438216 with bad export cookie 10667922797966238247 [ 1269.553058] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1269.588214] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.33@tcp (stopping) [ 1273.847186] LustreError: 42534:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1273.854597] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1273.893080] LustreError: 42534:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1273.943661] Lustre: Skipped 2 previous similar messages [ 1273.958504] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1276.044845] Lustre: server umount lustre-MDT0001 complete [ 1290.968786] Lustre: server umount lustre-OST0000 complete [ 1305.380657] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 1307.851896] Key type lgssc unregistered [ 1308.173181] LNet: 43813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1308.179909] LNetError: 43813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1309.225391] LNet: Removed LNI 192.168.201.133@tcp [ 1310.264422] Key type .llcrypt unregistered [ 1310.271739] Key type ._llcrypt unregistered [ 1332.770651] Key type ._llcrypt registered [ 1332.772879] Key type .llcrypt registered [ 1332.831560] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 1333.386859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1333.425185] alg: No test for adler32 (adler32-zlib) [ 1334.411812] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1334.616915] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 1336.272410] Key type lgssc registered [ 1337.299709] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1346.791820] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1346.805751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1348.417151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1352.804666] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1357.797059] LustreError: 45225:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1357.816207] LustreError: 45225:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1362.892494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1362.912522] LustreError: 45226:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1363.335181] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1369.122859] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1377.985045] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1384.302634] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1391.627858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1392.046796] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1393.084874] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1397.835471] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1406.308195] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1418.591467] Lustre: lustre-OST0000: Not available for connect from 192.168.201.33@tcp (stopping) [ 1418.724975] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1418.733908] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1418.746713] LustreError: Skipped 1 previous similar message [ 1422.020494] Lustre: server umount lustre-OST0000 complete [ 1440.227427] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1440.229899] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1440.247622] Lustre: Skipped 1 previous similar message [ 1440.249608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1440.254399] Lustre: Skipped 2 previous similar messages [ 1444.320262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1444.321578] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1446.562966] Lustre: server umount lustre-MDT0000 complete [ 1449.444690] LustreError: 45221:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1449.478067] LustreError: 45221:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1450.110474] LustreError: 45208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438397 with bad export cookie 5517630274764664010 [ 1450.124497] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1450.126698] LustreError: 45208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1456.567946] Lustre: server umount lustre-MDT0001 complete [ 1469.300179] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 1472.171591] Key type lgssc unregistered [ 1472.524372] LNet: 48935:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1472.538769] LNetError: 48935:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1472.571051] LNet: Removed LNI 192.168.201.133@tcp [ 1473.725825] Key type .llcrypt unregistered [ 1473.732296] Key type ._llcrypt unregistered [ 1501.275476] Key type ._llcrypt registered [ 1501.278327] Key type .llcrypt registered [ 1501.443159] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 1502.616120] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1502.808278] alg: No test for adler32 (adler32-zlib) [ 1503.901163] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1504.245192] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 1505.959174] Key type lgssc registered [ 1507.085892] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1515.273536] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1515.288091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1516.786321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1521.308422] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1526.250806] LustreError: 50345:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1526.280798] LustreError: 50345:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1530.374966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1530.770390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1536.216976] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1545.023426] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1551.339166] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1558.629929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1558.950729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1561.013499] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1566.511889] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1575.946272] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1583.030352] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1587.552579] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1600.354723] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-03 8h29m04s [ 1602.337195] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1606.116533] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1606.142361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1606.144761] Lustre: Skipped 1 previous similar message [ 1607.202935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1609.547327] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.33@tcp (stopping) [ 1609.557320] Lustre: Skipped 2 previous similar messages [ 1611.356630] Lustre: server umount lustre-MDT0000 complete [ 1613.294767] LustreError: 50340:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1613.325920] LustreError: 50340:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1615.521878] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1616.706121] LustreError: 50341:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1619.440181] LustreError: 50345:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1619.447813] LustreError: 50345:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1623.380778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1623.601642] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1623.941397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1624.085157] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1624.939391] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1629.379688] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1633.413376] Lustre: DEBUG MARKER: Wait for df (36715) ... [ 1635.374370] Lustre: DEBUG MARKER: done [ 1640.418865] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1640.437271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1640.440083] Lustre: Skipped 3 previous similar messages [ 1640.452082] Lustre: Skipped 2 previous similar messages [ 1645.555775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1645.558410] Lustre: Skipped 1 previous similar message [ 1646.589581] Lustre: server umount lustre-OST0000 complete [ 1655.777300] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1655.784366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1655.785217] Lustre: Skipped 1 previous similar message [ 1655.798477] Lustre: Skipped 1 previous similar message [ 1657.798223] Lustre: server umount lustre-MDT0000 complete [ 1662.101517] LustreError: 52972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438609 with bad export cookie 2810378791486871518 [ 1662.102812] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1662.112859] LustreError: 52972:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1662.734450] Lustre: server umount lustre-MDT0001 complete [ 1676.219900] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 1679.312499] Key type lgssc unregistered [ 1679.575285] LNet: 55273:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1679.582859] LNetError: 55273:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1679.594990] LNet: Removed LNI 192.168.201.133@tcp [ 1680.333729] Key type .llcrypt unregistered [ 1680.341427] Key type ._llcrypt unregistered [ 1725.307768] Key type ._llcrypt registered [ 1725.309505] Key type .llcrypt registered [ 1725.408930] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 1726.275928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1726.426455] alg: No test for adler32 (adler32-zlib) [ 1727.834222] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1728.075431] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 1729.743300] Key type lgssc registered [ 1731.063379] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1738.988222] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1739.016139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1740.440906] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1740.516113] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1740.708250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1746.255779] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1757.886697] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1757.946835] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1757.970962] Lustre: 57478:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1758.165166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1763.919314] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1771.830855] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1777.835885] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1785.258327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1785.494487] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1785.730097] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1792.730413] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1794.228813] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 1803.486151] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1811.872484] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-03 8h32m36s [ 1813.988810] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1822.437791] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1825.352237] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1825.473505] LustreError: 58692:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1845.635523] Lustre: lustre-MDT0000: Client 696249a5-bc72-4ab0-bf9d-68c2f6e89d8d (at 192.168.201.33@tcp) reconnecting [ 1865.551192] LustreError: 58692:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1868.403983] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1871.194873] Lustre: DEBUG MARKER: done [ 1879.525075] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1879.535221] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1879.556276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1880.038634] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1880.043151] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1880.075228] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1880.080473] Lustre: Skipped 1 previous similar message [ 1884.479836] Lustre: server umount lustre-OST0000 complete [ 1890.280943] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1890.314523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1896.886150] Lustre: server umount lustre-MDT0000 complete [ 1897.444364] LustreError: 57488:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1897.471504] LustreError: 57488:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1901.306373] LustreError: 57469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788438848 with bad export cookie 1325337037931052458 [ 1901.311108] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1901.324529] LustreError: 57469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1901.660705] Lustre: server umount lustre-MDT0001 complete [ 1915.011960] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 1918.240264] Key type lgssc unregistered [ 1918.663773] LNet: 61810:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1918.674701] LNetError: 61810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1918.697412] LNet: Removed LNI 192.168.201.133@tcp [ 1919.824622] Key type .llcrypt unregistered [ 1919.826892] Key type ._llcrypt unregistered [ 1969.087620] Key type ._llcrypt registered [ 1969.090291] Key type .llcrypt registered [ 1969.211666] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 1970.761236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1970.785050] alg: No test for adler32 (adler32-zlib) [ 1971.957390] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 1972.268809] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 1973.975858] Key type lgssc registered [ 1974.988415] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1978.670425] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1978.689924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1980.026861] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1980.059301] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1980.280156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1981.370892] Lustre: Failing over lustre-MDT0000 [ 1981.685542] Lustre: server umount lustre-MDT0000 complete [ 1989.766206] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 08:35:34 (1788438934) [ 1997.168937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1997.908687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2002.899916] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2014.147588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2014.267252] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2014.333942] Lustre: 64606:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2014.614215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2020.646840] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2029.345909] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2036.133898] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2042.595860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2042.728878] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2042.882523] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2049.445382] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2051.338986] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2057.508379] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2063.789997] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2071.008892] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2071.015151] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2071.029261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2072.034705] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2072.040135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2072.067964] Lustre: Skipped 1 previous similar message [ 2072.090818] Lustre: Skipped 2 previous similar messages [ 2077.130969] Lustre: server umount lustre-MDT0000 complete [ 2081.059861] LustreError: 66462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439028 with bad export cookie 13198781029448850151 [ 2081.065416] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2081.070462] LustreError: 66462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2081.420881] Lustre: server umount lustre-MDT0001 complete [ 2084.748139] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2089.659561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2097.434225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2098.655297] Lustre: 63529:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788439029/real 1788439029] req@000000006d918686 x1875314127971328/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788439045 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2098.706283] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2101.791194] Lustre: 63529:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788439033/real 1788439033] req@00000000aa98a33c x1875314127971456/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788439049 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2103.775510] Lustre: 63532:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788439034/real 1788439034] req@00000000702cf329 x1875314127971584/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788439050 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2105.891621] LustreError: 63528:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000009825576b x1875314127972352/t0(0) o250->MGC192.168.201.133@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2106.388081] LustreError: 66455:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2106.417078] LustreError: 66455:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2106.523732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2106.614293] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2107.488208] Lustre: 63532:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788439038/real 1788439038] req@000000006fce2b04 x1875314127971840/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788439054 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2108.579361] LustreError: 66456:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2108.614566] LustreError: 66456:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2112.481630] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2112.672241] Lustre: 63529:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788439043/real 1788439043] req@00000000916df2bd x1875314127971968/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788439059 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2114.031808] LustreError: 66455:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2114.063617] LustreError: 66455:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2119.137933] LustreError: 68379:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2119.170799] LustreError: 68379:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2122.996447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2123.355653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2127.468102] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2127.993681] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2136.474716] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2143.704108] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2150.057651] Lustre: server umount lustre-MDT0000 complete [ 2153.272290] LustreError: 64600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439100 with bad export cookie 13198781029448852426 [ 2153.293357] LustreError: 64600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2153.294714] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2153.446686] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2153.457133] LustreError: 69867:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2153.460408] Lustre: Skipped 1 previous similar message [ 2153.487616] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2155.647102] Lustre: server umount lustre-MDT0001 complete [ 2161.886884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2171.472143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2178.016155] LustreError: 63528:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000007aeef007 x1875314127999104/t0(0) o250->MGC192.168.201.133@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2178.247688] LustreError: 69113:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2178.341540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2178.392039] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2182.459297] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2190.919466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2193.491731] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2193.516727] Lustre: Skipped 1 previous similar message [ 2196.356313] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2203.429375] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2209.181540] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2216.929857] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2216.941532] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2216.961975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2222.370110] Lustre: server umount lustre-MDT0000 complete [ 2224.099789] LustreError: 71060:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2224.109121] LustreError: 71060:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2226.088730] LustreError: 64599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439173 with bad export cookie 13198781029448853805 [ 2226.091643] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2226.100024] LustreError: 64599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2226.373314] Lustre: server umount lustre-MDT0001 complete [ 2232.135801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2236.032488] Lustre: DEBUG MARKER: files compared the same [ 2248.356686] Lustre: server umount lustre-OST0000 complete [ 2265.195628] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 2267.593746] Key type lgssc unregistered [ 2267.849593] LNet: 74336:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2267.867273] LNetError: 74336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2268.910114] LNet: Removed LNI 192.168.201.133@tcp [ 2269.713954] Key type .llcrypt unregistered [ 2269.715977] Key type ._llcrypt unregistered [ 2292.527233] Key type ._llcrypt registered [ 2292.530319] Key type .llcrypt registered [ 2292.632528] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 2293.736140] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2293.863741] alg: No test for adler32 (adler32-zlib) [ 2295.039677] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2295.283833] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 2296.919930] Key type lgssc registered [ 2298.081498] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2306.088883] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2306.111406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2307.499733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2312.289248] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug +malloc all [ 2316.791715] LustreError: 75752:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2316.813971] LustreError: 75752:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2320.610380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2320.906428] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2324.013878] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug +malloc all [ 2331.150723] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2339.036982] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2348.838936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2349.271307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2352.413755] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2357.420732] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug +malloc all [ 2366.834709] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2377.189489] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2377.194291] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2377.242485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2382.304471] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2382.308858] Lustre: Skipped 1 previous similar message [ 2382.899686] Lustre: server umount lustre-OST0000 complete [ 2387.428259] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2387.436332] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2387.456581] Lustre: Skipped 1 previous similar message [ 2387.458341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2392.550663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2392.555479] Lustre: Skipped 2 previous similar messages [ 2393.037533] Lustre: server umount lustre-MDT0000 complete [ 2397.157130] LustreError: 75731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439344 with bad export cookie 11304046017206043670 [ 2397.164782] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2397.171787] LustreError: 75731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2397.540661] Lustre: server umount lustre-MDT0001 complete [ 2409.495048] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 2412.448707] Key type lgssc unregistered [ 2412.817529] LNet: 79286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2412.841494] LNetError: 79286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2413.881136] LNet: Removed LNI 192.168.201.133@tcp [ 2415.108599] Key type .llcrypt unregistered [ 2415.112675] Key type ._llcrypt unregistered [ 2438.796549] Key type ._llcrypt registered [ 2438.799779] Key type .llcrypt registered [ 2438.924263] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 2440.336475] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2440.415187] alg: No test for adler32 (adler32-zlib) [ 2441.501744] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2441.771234] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 2443.439157] Key type lgssc registered [ 2444.562822] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2452.387659] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2452.418156] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2468.127446] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2473.247680] LustreError: 80668:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.133@tcp: failed processing log, type 4: rc = -110 [ 2499.935351] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2505.619563] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2514.135750] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2522.127792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2522.273702] Lustre: *** cfs_fail_loc=706, val=0*** [ 2522.697299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2522.770517] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2527.503517] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2527.716458] LustreError: 80694:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2527.742109] LustreError: 80694:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2532.848464] LustreError: 80693:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2533.876086] LustreError: 81754:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2533.895421] LustreError: 81754:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2536.350440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2536.727026] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2542.071268] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2549.707466] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2555.391276] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2558.432457] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2558.438385] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2558.448854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2562.532305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2562.533535] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2562.541198] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2567.651763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2567.664143] Lustre: Skipped 2 previous similar messages [ 2572.255112] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2572.427898] Lustre: server umount lustre-OST0000 complete [ 2577.888336] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2577.889905] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2577.911768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2577.927794] Lustre: Skipped 1 previous similar message [ 2583.008900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2583.013815] Lustre: Skipped 1 previous similar message [ 2590.175866] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2590.395982] Lustre: server umount lustre-MDT0000 complete [ 2593.256736] LustreError: 81758:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2593.276696] LustreError: 81758:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2594.263468] LustreError: 80674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788439541 with bad export cookie 10982101294863125892 [ 2594.275267] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2594.281330] LustreError: 80674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2594.519708] Lustre: server umount lustre-MDT0001 complete [ 2606.338122] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 2609.038397] Key type lgssc unregistered [ 2609.362666] LNet: 84283:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2609.374131] LNetError: 84283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2609.394786] LNet: Removed LNI 192.168.201.133@tcp [ 2610.340815] Key type .llcrypt unregistered [ 2610.343099] Key type ._llcrypt unregistered [ 2636.756340] Key type ._llcrypt registered [ 2636.757879] Key type .llcrypt registered [ 2636.876969] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 2638.383384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2638.448372] alg: No test for adler32 (adler32-zlib) [ 2639.556766] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 2639.860472] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 2641.535210] Key type lgssc registered [ 2643.083213] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2651.131569] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2651.169456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2656.516714] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2666.381809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2667.034039] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2672.105508] LustreError: 86388:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2672.124138] LustreError: 86388:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2672.646387] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2677.224797] LustreError: 86392:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2677.260414] LustreError: 86392:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2682.341119] LustreError: 86388:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2682.360845] LustreError: 86388:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2682.755864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2683.148806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2688.486765] LustreError: 86388:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2688.501905] LustreError: 86388:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2689.705773] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2692.583965] LustreError: 87416:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2692.625394] LustreError: 87416:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2698.452366] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2698.461286] Lustre: MGS: Client 460b3642-96ce-4a92-873d-4f98957d735a (at 0@lo) reconnecting [ 2698.481795] LustreError: 86402:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2698.641574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2698.716567] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2703.454789] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2719.201372] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2719.208738] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2719.209615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2719.218683] Lustre: Skipped 1 previous similar message [ 2719.230474] Lustre: Skipped 1 previous similar message [ 2723.145727] Lustre: server umount lustre-OST0000 complete [ 2729.440700] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2729.441686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2729.460325] Lustre: Skipped 1 previous similar message [ 2729.473129] Lustre: Skipped 1 previous similar message [ 2733.800189] Lustre: server umount lustre-MDT0000 complete [ 2734.564576] LustreError: 86388:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2734.578495] LustreError: 86388:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2739.165649] Lustre: server umount lustre-MDT0001 complete [ 2743.264829] Lustre: server umount MGS complete [ 2755.154614] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 08:48:20 (1788439700) [ 2786.589803] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 2796.723470] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 2835.966806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2841.437292] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2851.487920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2851.804921] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2851.916572] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2852.036310] Lustre: lustre-MDT0001: new disk, initializing [ 2852.166137] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2852.198800] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2852.213391] Lustre: 93558:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2852.225179] LustreError: 93558:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2857.293333] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2867.787640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2867.954742] Lustre: lustre-OST0000: new disk, initializing [ 2867.958938] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2867.964971] Lustre: 94325:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2868.027120] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2876.003669] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2879.076930] Lustre: 94781:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000dd4b439a x1875314828419584/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788439842 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2887.567626] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2896.126532] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2896.137170] Lustre: MGS: Client 8e26489f-cc08-496a-9f67-fe1fbf49f400 (at 0@lo) reconnecting [ 2896.159761] LustreError: 93544:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2896.240777] Lustre: 92828:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2896.298696] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2896.439419] Lustre: lustre-MDT0000: new disk, initializing [ 2896.773537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2896.815357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2896.853594] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2897.891124] Lustre: 95394:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000c8c0c0d9 x1875314828426496/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788439861 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2903.144682] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2904.183822] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 2906.613381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2906.626874] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2907.197299] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2931.183668] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2931.191851] Lustre: Skipped 1 previous similar message [ 2931.194754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2931.223086] Lustre: Skipped 1 previous similar message [ 2936.288703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2937.021325] Lustre: server umount lustre-OST0000 complete [ 2946.529368] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2946.531326] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2946.547762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2946.547771] Lustre: Skipped 1 previous similar message [ 2946.574274] Lustre: Skipped 1 previous similar message [ 2956.769045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2956.771610] Lustre: Skipped 3 previous similar messages [ 2957.279693] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2957.471450] Lustre: server umount lustre-MDT0000 complete [ 2966.843767] Lustre: server umount lustre-MDT0001 complete [ 2971.460320] Lustre: server umount MGS complete [ 2984.877418] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 08:52:10 (1788439930) [ 3003.539278] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 3006.245862] Key type lgssc unregistered [ 3006.615924] LNet: 97895:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3006.625572] LNetError: 97895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3006.638328] LNet: Removed LNI 192.168.201.133@tcp [ 3007.491568] Key type .llcrypt unregistered [ 3007.493532] Key type ._llcrypt unregistered [ 3026.294403] Key type ._llcrypt registered [ 3026.299527] Key type .llcrypt registered [ 3026.494901] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 3027.470716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3027.484870] alg: No test for adler32 (adler32-zlib) [ 3028.549711] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3028.759292] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 3030.463713] Key type lgssc registered [ 3031.715941] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3040.845819] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3040.882460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3042.322674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3047.130714] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3052.520893] LustreError: 99120:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3052.529571] LustreError: 99120:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3056.004099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3056.326183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3062.269271] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3070.611794] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3077.163792] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3086.021857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3086.303378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3087.344580] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 3095.439306] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3104.907709] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3117.034664] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3117.035575] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3117.058111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3117.063711] Lustre: Skipped 1 previous similar message [ 3117.077875] Lustre: Skipped 1 previous similar message [ 3120.069718] Lustre: server umount lustre-OST0000 complete [ 3127.265132] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3127.273543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3127.276902] Lustre: Skipped 1 previous similar message [ 3127.282317] Lustre: Skipped 1 previous similar message [ 3129.772404] Lustre: server umount lustre-MDT0000 complete [ 3132.391281] LustreError: 99848:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3132.413121] LustreError: 99848:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3134.256516] LustreError: 100966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440081 with bad export cookie 2627487183638674017 [ 3134.268815] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3134.280506] LustreError: 100966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3134.882106] Lustre: server umount lustre-MDT0001 complete [ 3149.511901] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 3152.659259] Key type lgssc unregistered [ 3152.927810] LNet: 102653:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3152.942739] LNetError: 102653:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3153.963819] LNet: Removed LNI 192.168.201.133@tcp [ 3154.792348] Key type .llcrypt unregistered [ 3154.794078] Key type ._llcrypt unregistered [ 3172.051513] Key type ._llcrypt registered [ 3172.058139] Key type .llcrypt registered [ 3172.181705] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 3173.900980] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3173.930374] alg: No test for adler32 (adler32-zlib) [ 3175.070317] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3175.448580] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 3177.135216] Key type lgssc registered [ 3178.636924] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3186.222496] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3186.229605] LustreError: 103737:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 3186.249474] LustreError: 103740:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 3186.267492] LustreError: 103737:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4978 [ 3186.280460] LustreError: 103737:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3186.286813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3186.293195] LustreError: 103737:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3187.838745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3196.752124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3197.034358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3201.753630] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3207.107764] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing lsmod [ 3210.124869] LustreError: 104932:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 3210.246790] LustreError: 104948:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 3210.252565] LustreError: 104932:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4881 [ 3210.265271] LustreError: 104932:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3210.279725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3210.288767] LustreError: 104932:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3210.860055] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3212.929996] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 3217.378475] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3217.386687] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3217.409336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3218.407543] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3218.424379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3219.948188] Lustre: server umount lustre-MDT0000 complete [ 3222.562275] LustreError: 103778:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3222.603124] LustreError: 103778:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3224.449152] LustreError: 103758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440171 with bad export cookie 11887903918697967798 [ 3224.462462] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3224.465546] LustreError: 103758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3224.623220] LustreError: 104210:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000000adc2c3a x1875315389387776/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3225.207507] Lustre: server umount lustre-MDT0001 complete [ 3240.722376] Lustre: server umount lustre-OST0000 complete [ 3252.669992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3253.651483] LustreError: 106011:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3253.841713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3258.865672] LustreError: 106012:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3260.056763] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3262.951910] LustreError: 106011:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3268.076609] LustreError: 106012:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3268.819664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3269.163126] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3274.223081] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3283.666342] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3290.343868] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3297.718947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3298.138513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3299.218114] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:97) [ 3306.040991] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3314.933977] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3324.897300] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3324.900148] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3324.910328] LustreError: Skipped 1 previous similar message [ 3324.915454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3324.915466] Lustre: Skipped 2 previous similar messages [ 3324.966862] Lustre: Skipped 2 previous similar messages [ 3329.294798] Lustre: server umount lustre-OST0000 complete [ 3335.137906] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3335.139292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3335.147991] Lustre: Skipped 1 previous similar message [ 3335.151373] Lustre: Skipped 1 previous similar message [ 3340.004503] Lustre: server umount lustre-MDT0000 complete [ 3340.257423] LustreError: 106011:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3340.286316] LustreError: 106011:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3344.207094] LustreError: 105991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440291 with bad export cookie 11887903918697968729 [ 3344.214985] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3344.223712] LustreError: 105991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3344.618494] Lustre: server umount lustre-MDT0001 complete [ 3357.901365] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 3361.726495] Key type lgssc unregistered [ 3362.112795] LNet: 109545:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3362.120462] LNetError: 109545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3362.136451] LNet: Removed LNI 192.168.201.133@tcp [ 3363.085709] Key type .llcrypt unregistered [ 3363.088248] Key type ._llcrypt unregistered [ 3391.907270] Key type ._llcrypt registered [ 3391.915291] Key type .llcrypt registered [ 3392.093844] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 3394.044273] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3394.117803] alg: No test for adler32 (adler32-zlib) [ 3395.152184] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3395.349431] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 3397.015165] Key type lgssc registered [ 3398.537793] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3407.391962] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3407.411185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3408.884324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3414.156679] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3419.108644] LustreError: 110959:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3419.137939] LustreError: 110959:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3424.226701] LustreError: 110960:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3424.653622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3425.228895] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3430.364803] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3439.198229] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3445.826681] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3454.191470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3454.666416] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3462.173862] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3470.909448] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3479.277676] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3488.246650] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3492.982551] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3500.002991] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3500.005097] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3500.005990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3500.035590] Lustre: Skipped 1 previous similar message [ 3505.138843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3505.141817] Lustre: Skipped 1 previous similar message [ 3510.249389] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3510.258823] Lustre: Skipped 2 previous similar messages [ 3512.287862] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3512.569792] Lustre: server umount lustre-OST0000 complete [ 3517.409216] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3517.413615] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3517.423080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3520.493677] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3523.638165] Lustre: server umount lustre-MDT0000 complete [ 3525.601297] LustreError: 110959:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3525.624939] LustreError: 110959:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3527.577174] LustreError: 110940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440474 with bad export cookie 18352999357409871164 [ 3527.584140] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3527.586073] LustreError: 110940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3527.921789] Lustre: server umount lustre-MDT0001 complete [ 3539.072454] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 3542.192455] Key type lgssc unregistered [ 3542.527128] LNet: 114635:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3542.533750] LNetError: 114635:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3542.580182] LNet: Removed LNI 192.168.201.133@tcp [ 3543.353322] Key type .llcrypt unregistered [ 3543.355395] Key type ._llcrypt unregistered [ 3561.958864] Key type ._llcrypt registered [ 3561.960915] Key type .llcrypt registered [ 3562.101852] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 3563.439656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3563.607565] alg: No test for adler32 (adler32-zlib) [ 3564.726874] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3565.053427] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 3566.807318] Key type lgssc registered [ 3568.017367] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3576.484651] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3576.502708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3578.066675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3584.254297] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3588.580856] LustreError: 115855:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3588.595442] LustreError: 115855:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3593.700651] LustreError: 115856:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3593.897575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3594.313756] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3599.339165] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3607.644824] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3613.770976] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3620.972318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3621.388244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3623.438514] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3628.212211] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3636.316334] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3650.029112] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3650.030049] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3650.040647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3650.043063] Lustre: Skipped 1 previous similar message [ 3655.194771] LustreError: 117675:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3655.206040] LustreError: 117675:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3655.314755] Lustre: server umount lustre-OST0000 complete [ 3660.257442] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3660.270814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3660.273404] Lustre: Skipped 1 previous similar message [ 3665.377318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3665.387698] Lustre: Skipped 1 previous similar message [ 3665.919915] Lustre: server umount lustre-MDT0000 complete [ 3670.110526] LustreError: 115836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440617 with bad export cookie 2934034450198161481 [ 3670.120099] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3670.121229] LustreError: 115836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3670.396163] Lustre: server umount lustre-MDT0001 complete [ 3682.037440] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 3684.549694] Key type lgssc unregistered [ 3684.781737] LNet: 119399:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3684.791059] LNetError: 119399:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3685.865883] LNet: Removed LNI 192.168.201.133@tcp [ 3686.698363] Key type .llcrypt unregistered [ 3686.700224] Key type ._llcrypt unregistered [ 3712.404822] Key type ._llcrypt registered [ 3712.410449] Key type .llcrypt registered [ 3712.561347] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 3713.856802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3714.006105] alg: No test for adler32 (adler32-zlib) [ 3715.007853] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3715.217035] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 3716.959340] Key type lgssc registered [ 3718.090473] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3726.733921] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3726.761049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3728.266385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3733.103674] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3738.611369] LustreError: 120824:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3738.626422] LustreError: 120824:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3742.569063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3743.029461] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3748.594797] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3756.516883] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3763.307324] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3770.346627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3770.762334] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3773.814672] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:163 to 0x240000401:193) [ 3777.592757] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3785.961301] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3794.113427] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3798.638156] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3798.648367] Lustre: Skipped 2 previous similar messages [ 3803.851102] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3803.854306] Lustre: Skipped 2 previous similar messages [ 3804.273773] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3804.319284] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3809.860711] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.33@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3809.866525] Lustre: Skipped 2 previous similar messages [ 3814.530477] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.33@tcp 192.168.0.[2,10]@tcp [ 3819.029745] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.33/24@tcp in log lustre-MDT0000 [ 3819.033825] Lustre: Skipped 2 previous similar messages [ 3822.670852] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.33/24@tcp [ 3822.675331] Lustre: Skipped 1 previous similar message [ 3829.220230] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3829.221409] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3829.244984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3829.249763] Lustre: Skipped 1 previous similar message [ 3834.340932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3834.358729] Lustre: Skipped 1 previous similar message [ 3835.162202] Lustre: server umount lustre-OST0000 complete [ 3839.456890] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3839.468111] Lustre: Skipped 1 previous similar message [ 3839.474754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3839.485615] Lustre: Skipped 1 previous similar message [ 3844.576596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3844.594341] Lustre: Skipped 1 previous similar message [ 3845.507283] Lustre: server umount lustre-MDT0000 complete [ 3848.877324] LustreError: 121563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440796 with bad export cookie 7264239745425552723 [ 3848.886020] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3848.887986] LustreError: 121563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3849.463941] Lustre: server umount lustre-MDT0001 complete [ 3863.307456] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 3867.611047] Key type lgssc unregistered [ 3867.939736] LNet: 125567:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3867.957812] LNetError: 125567:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3868.028954] LNet: Removed LNI 192.168.201.133@tcp [ 3869.454976] Key type .llcrypt unregistered [ 3869.458984] Key type ._llcrypt unregistered [ 3900.086525] Key type ._llcrypt registered [ 3900.089188] Key type .llcrypt registered [ 3900.193910] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 3901.101258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3901.109690] alg: No test for adler32 (adler32-zlib) [ 3902.140913] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 3902.394108] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 3904.073536] Key type lgssc registered [ 3905.497608] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3914.015345] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3914.056113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3915.644061] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3915.655280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3915.686587] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.33/24@tcp [ 3920.298391] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3924.963437] LustreError: 127106:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3924.997739] LustreError: 127106:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3929.597468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3929.899437] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3929.929854] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.33/24@tcp [ 3929.993990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3934.432785] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3942.221795] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3948.998732] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3956.092102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3956.336045] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3960.446807] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:201 to 0x240000401:225) [ 3963.752962] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 3973.163594] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3991.526748] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3991.537947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3991.547416] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3991.566384] Lustre: Skipped 1 previous similar message [ 3991.607333] Lustre: Skipped 1 previous similar message [ 3993.023356] Lustre: server umount lustre-OST0000 complete [ 4001.762052] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4001.762779] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4001.773493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4001.784907] LustreError: Skipped 1 previous similar message [ 4004.342321] Lustre: server umount lustre-MDT0000 complete [ 4006.881231] LustreError: 127107:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4006.892546] LustreError: 127107:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4008.746218] LustreError: 127086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788440955 with bad export cookie 11109996423848701160 [ 4008.765973] LustreError: 127086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4008.768283] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4009.151639] Lustre: server umount lustre-MDT0001 complete [ 4021.610969] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 4024.868384] Key type lgssc unregistered [ 4025.097016] LNet: 130694:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4025.103332] LNetError: 130694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4026.152950] LNet: Removed LNI 192.168.201.133@tcp [ 4027.227562] Key type .llcrypt unregistered [ 4027.234646] Key type ._llcrypt unregistered [ 4056.995463] Key type ._llcrypt registered [ 4056.998277] Key type .llcrypt registered [ 4057.117993] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 4072.512689] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 4074.077208] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4074.245465] alg: No test for adler32 (adler32-zlib) [ 4075.558573] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 4075.970808] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 4077.759171] Key type lgssc registered [ 4079.107369] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4121.292977] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4121.350503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4122.711234] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4122.773964] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4122.873874] Lustre: lustre-MDT0000: new disk, initializing [ 4123.009113] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4123.057369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4129.468420] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4141.072790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4141.189205] Lustre: 134094:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4141.271264] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4141.274853] Lustre: Skipped 1 previous similar message [ 4141.350646] Lustre: lustre-MDT0001: new disk, initializing [ 4141.467342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4141.501325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4141.513282] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4146.619950] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4156.372910] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4162.907412] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4169.853873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4170.169700] Lustre: lustre-OST0000: new disk, initializing [ 4170.173343] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4170.181822] Lustre: 136064:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4170.335690] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4177.525204] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4178.528030] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4178.546112] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4178.741614] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4186.860252] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4191.448805] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4191.653790] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4195.934829] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4196.238586] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4199.393372] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4199.398413] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4199.406799] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4199.430311] Lustre: Skipped 1 previous similar message [ 4204.379368] Lustre: server umount lustre-OST0000 complete [ 4209.636227] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4209.655918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4209.659913] Lustre: Skipped 1 previous similar message [ 4209.678600] Lustre: Skipped 2 previous similar messages [ 4216.830074] Lustre: server umount lustre-MDT0000 complete [ 4220.615287] LustreError: 134087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441167 with bad export cookie 2440343851577888683 [ 4220.622100] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4220.625604] LustreError: 134087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4221.157728] Lustre: server umount lustre-MDT0001 complete [ 4231.589679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4232.312238] LustreError: 138281:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4232.323944] LustreError: 138281:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4232.356651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4237.798550] LustreError: 138282:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4238.527027] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4241.956442] LustreError: 138281:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4247.011127] LustreError: 138282:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4250.235298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4250.672910] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4256.226641] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4265.885095] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4272.813261] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4280.416957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4280.665810] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4287.356657] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4296.350493] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4301.559739] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4303.850237] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 4308.375710] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4308.610725] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4318.100722] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4318.307540] Lustre: lustre-OST0001: new disk, initializing [ 4318.310323] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4318.315100] Lustre: 141668:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4318.385204] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4323.973665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4323.988851] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4324.158371] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4325.954554] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4335.680759] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4335.967103] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4340.715534] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4341.097108] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4347.202820] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff939cd8121000.ost_server_uuid 50 [ 4349.168286] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff939cd8121000.ost_server_uuid in FULL state after 0 sec [ 4360.168529] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4360.195623] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4360.197770] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4360.202256] LustreError: Skipped 1 previous similar message [ 4360.217032] Lustre: Skipped 1 previous similar message [ 4365.295071] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4365.313957] Lustre: Skipped 1 previous similar message [ 4365.856290] Lustre: server umount lustre-OST0001 complete [ 4370.414378] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4370.417270] LustreError: 140105:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4370.427446] Lustre: Skipped 2 previous similar messages [ 4370.429920] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4370.455827] LustreError: 140105:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4376.766630] Lustre: server umount lustre-OST0000 complete [ 4383.714055] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4383.726221] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4383.739074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4383.745224] Lustre: Skipped 3 previous similar messages [ 4387.588583] Lustre: server umount lustre-MDT0000 complete [ 4390.889993] LustreError: 139010:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4390.929921] LustreError: 139010:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4392.063088] LustreError: 138263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441339 with bad export cookie 2440343851577889565 [ 4392.068876] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4392.075071] LustreError: 138263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4392.799101] Lustre: server umount lustre-MDT0001 complete [ 4414.925099] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 4418.222359] Key type lgssc unregistered [ 4418.528525] LNet: 144672:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4418.540689] LNetError: 144672:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4418.554417] LNet: Removed LNI 192.168.201.133@tcp [ 4419.440752] Key type .llcrypt unregistered [ 4419.447916] Key type ._llcrypt unregistered [ 4462.924432] Key type ._llcrypt registered [ 4462.926595] Key type .llcrypt registered [ 4463.014571] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 4478.826432] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 4480.163627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4480.175348] alg: No test for adler32 (adler32-zlib) [ 4481.271585] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 4481.574554] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 4483.297537] Key type lgssc registered [ 4485.122099] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4525.911876] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4525.951761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4527.592968] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4527.638656] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4527.826882] Lustre: lustre-MDT0000: new disk, initializing [ 4527.960975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4528.018361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4534.138735] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4549.076258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4549.350921] Lustre: 148831:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4549.560129] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4549.565633] Lustre: Skipped 1 previous similar message [ 4549.742060] Lustre: lustre-MDT0001: new disk, initializing [ 4549.870395] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4549.922337] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4549.944922] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4555.707806] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4568.432754] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4575.520795] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4583.540772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4583.805781] Lustre: lustre-OST0000: new disk, initializing [ 4583.808608] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4583.823092] Lustre: 150801:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4583.930952] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4591.210575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4591.232897] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4591.412531] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4591.702207] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4602.727521] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4608.650061] Lustre: Failing over lustre-OST0000 [ 4608.893250] Lustre: server umount lustre-OST0000 complete [ 4626.403407] Lustre: 147049:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788441557/real 1788441557] req@00000000c5647095 x1875316758842752/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788441573 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4626.440109] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4627.938498] Lustre: 147050:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788441559/real 1788441559] req@000000005858081c x1875316758843776/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788441575 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4627.986440] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4629.090793] Lustre: 147049:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788441559/real 1788441559] req@00000000cdb8d389 x1875316758843648/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788441575 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4630.023410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4630.355297] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4630.375867] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4633.119139] Lustre: 147052:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788441564/real 1788441564] req@0000000025fd2e05 x1875316758845056/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788441580 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4633.153460] Lustre: 147052:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4635.969404] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4638.175138] Lustre: 147050:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788441569/real 1788441569] req@0000000047a4ffaf x1875316758846336/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788441585 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4638.224651] Lustre: 147050:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4638.624040] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4638.625457] Lustre: lustre-OST0000: Recovery over after 0:03, of 3 clients 3 recovered and 0 were evicted. [ 4638.938520] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4643.525795] Lustre: Failing over lustre-MDT0000 [ 4643.891967] Lustre: server umount lustre-MDT0000 complete [ 4646.880947] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4651.324836] LustreError: 151054:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4651.352568] LustreError: 151054:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4652.000839] LustreError: 148842:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4652.025246] LustreError: 148842:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4655.011869] LustreError: 148837:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4655.036240] LustreError: 148837:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4660.194279] LustreError: 148837:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4660.237273] LustreError: 148837:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4663.268489] Lustre: 147052:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788441594/real 1788441594] req@000000006d918686 x1875316758860032/t0(0) o400->MGC192.168.201.133@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788441610 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4663.298770] Lustre: 147052:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4663.312734] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4665.258685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4666.687505] LustreError: 151054:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4666.731074] LustreError: 151054:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4673.504222] LustreError: 147048:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000000b77a16c x1875316758866176/t0(0) o250->MGC192.168.201.133@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4673.838688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4673.905657] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4677.954884] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4679.165559] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4679.178991] Lustre: Skipped 1 previous similar message [ 4679.211323] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 4679.273615] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4679.705504] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4689.378913] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4689.380371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4689.397138] Lustre: Skipped 3 previous similar messages [ 4689.409542] Lustre: Skipped 1 previous similar message [ 4694.512273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4694.516845] Lustre: Skipped 1 previous similar message [ 4695.266175] Lustre: server umount lustre-OST0000 complete [ 4704.747094] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4704.752254] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4704.771787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4704.787605] Lustre: Skipped 1 previous similar message [ 4705.986505] Lustre: server umount lustre-MDT0000 complete [ 4709.864694] LustreError: 148842:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4709.891781] LustreError: 148842:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 4710.160247] LustreError: 148823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788441657 with bad export cookie 12029400708587231798 [ 4710.165783] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4710.175400] LustreError: 148823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4710.698583] Lustre: server umount lustre-MDT0001 complete [ 4722.768093] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 4726.170079] Key type lgssc unregistered [ 4726.429800] LNet: 154457:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4726.437594] LNetError: 154457:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4727.474073] LNet: Removed LNI 192.168.201.133@tcp [ 4728.579634] Key type .llcrypt unregistered [ 4728.581731] Key type ._llcrypt unregistered [ 4770.643605] Key type ._llcrypt registered [ 4770.649843] Key type .llcrypt registered [ 4770.792623] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 4772.293632] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4772.343354] alg: No test for adler32 (adler32-zlib) [ 4773.864548] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 4774.226455] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 4776.007557] Key type lgssc registered [ 4778.029975] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4785.573174] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4785.588809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4787.158587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4792.812157] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4797.409502] LustreError: 155976:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4797.430781] LustreError: 155976:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4802.529229] LustreError: 155975:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4803.680675] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4807.651621] LustreError: 155976:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4808.305950] LustreError: 156004:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4808.315232] LustreError: 156004:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4808.322885] LustreError: 156004:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 4809.132850] Lustre: server umount lustre-MDT0000 complete [ 4827.689041] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 4839.465696] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 4883.317607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4883.490743] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4883.525532] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4883.642309] Lustre: lustre-MDT0000: new disk, initializing [ 4883.817070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4883.842580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4888.382593] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4900.757529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4900.872501] Lustre: 159310:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4900.920078] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4900.924487] Lustre: Skipped 1 previous similar message [ 4900.983466] Lustre: lustre-MDT0001: new disk, initializing [ 4901.109020] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4901.166264] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4901.180521] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4906.503977] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4917.277381] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4925.150518] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4932.582837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4932.842672] Lustre: lustre-OST0000: new disk, initializing [ 4932.846727] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4932.852586] Lustre: 161287:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4932.972544] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4938.840766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4938.860000] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4939.000369] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4940.901696] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 4951.899575] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4956.998694] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4959.116804] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 09:25:04 (1788441904) [ 5231.071656] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5231.094305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5236.193752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5236.202337] Lustre: Skipped 4 previous similar messages [ 5241.315475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5245.407151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5245.721376] Lustre: server umount lustre-MDT0000 complete [ 5246.436830] LustreError: 159315:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5246.462024] LustreError: 159315:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5250.615444] LustreError: 160112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442197 with bad export cookie 16256966731358790071 [ 5250.619546] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5250.626777] LustreError: 160112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5251.552310] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5251.554448] LustreError: 159314:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5251.568316] Lustre: Skipped 2 previous similar messages [ 5251.570575] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5251.570585] Lustre: Skipped 2 previous similar messages [ 5255.585409] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5257.293306] Lustre: server umount lustre-MDT0001 complete [ 5268.285978] Lustre: server umount lustre-OST0000 complete [ 5279.499684] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 5290.997459] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 5331.263320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5331.581447] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5331.601841] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5331.724638] Lustre: lustre-MDT0000: new disk, initializing [ 5331.808715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5331.824644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5337.337251] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 5348.881921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5349.064745] Lustre: 165954:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5349.126784] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5349.131143] Lustre: Skipped 1 previous similar message [ 5349.201277] Lustre: lustre-MDT0001: new disk, initializing [ 5349.339988] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5349.399512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5349.414261] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5354.620921] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 5363.684208] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5369.237958] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5377.699084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5378.017635] Lustre: lustre-OST0000: new disk, initializing [ 5378.024657] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5378.036710] Lustre: 167922:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5378.170086] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5386.305383] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 5386.890206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5386.919959] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5387.100353] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5397.736734] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5402.560406] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5402.776351] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5408.721982] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5408.945187] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5412.835438] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5412.842196] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5412.848173] LustreError: Skipped 1 previous similar message [ 5412.850391] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5412.863607] Lustre: Skipped 1 previous similar message [ 5412.881174] Lustre: Skipped 1 previous similar message [ 5417.349257] Lustre: server umount lustre-OST0000 complete [ 5423.074096] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5423.091805] Lustre: Skipped 1 previous similar message [ 5433.341551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5433.347106] Lustre: Skipped 5 previous similar messages [ 5436.383417] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5436.658601] Lustre: server umount lustre-MDT0000 complete [ 5438.435432] LustreError: 165961:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5438.466118] LustreError: 165961:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5440.815654] LustreError: 166749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442388 with bad export cookie 16256966731358948901 [ 5440.830031] LustreError: 166749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5440.833662] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5441.147675] Lustre: server umount lustre-MDT0001 complete [ 5450.782977] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 09:33:16 (1788442396) [ 5467.518960] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 5477.181885] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 5518.746029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5519.121969] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5519.179952] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5519.299149] Lustre: lustre-MDT0000: new disk, initializing [ 5519.428628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5519.461669] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5524.675662] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 5535.531650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5535.662657] Lustre: 172680:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5536.026190] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5540.983178] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 5549.760687] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5555.667920] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5562.671816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5562.882941] Lustre: lustre-OST0000: new disk, initializing [ 5562.891575] Lustre: Skipped 1 previous similar message [ 5562.901687] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5562.905797] Lustre: Skipped 2 previous similar messages [ 5562.915683] Lustre: 174659:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5569.762980] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 5570.647281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5570.654822] Lustre: Skipped 1 previous similar message [ 5570.667945] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5570.879205] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5579.213709] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5586.406602] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5586.411362] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5586.416993] LustreError: Skipped 1 previous similar message [ 5586.434666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5600.223173] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5600.503555] Lustre: server umount lustre-OST0000 complete [ 5606.888475] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5606.911158] Lustre: Skipped 2 previous similar messages [ 5611.134610] Lustre: server umount lustre-MDT0000 complete [ 5612.011475] LustreError: 175564:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5615.026904] LustreError: 173481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442562 with bad export cookie 16256966731358949783 [ 5615.037649] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5615.039401] LustreError: 173481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5615.453115] Lustre: server umount lustre-MDT0001 complete [ 5624.534815] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 09:36:09 (1788442569) [ 5641.520296] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 5652.934620] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 5690.855301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5691.135712] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5691.158527] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5691.212659] Lustre: lustre-MDT0000: new disk, initializing [ 5691.269746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5691.274474] Lustre: Skipped 2 previous similar messages [ 5691.287205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5695.892868] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 5706.078853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5706.185890] Lustre: 178984:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5706.561345] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5711.220295] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 5721.193726] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5727.805664] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5737.311511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5737.902443] Lustre: 180961:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5743.793758] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5744.029117] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5747.237358] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 5760.378799] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5769.707686] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5769.708707] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5769.711629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5769.711638] Lustre: Skipped 7 previous similar messages [ 5769.733907] Lustre: Skipped 1 previous similar message [ 5769.783800] LustreError: Skipped 1 previous similar message [ 5775.353418] Lustre: server umount lustre-OST0000 complete [ 5786.056327] Lustre: server umount lustre-MDT0000 complete [ 5790.156210] LustreError: 178976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442737 with bad export cookie 16256966731358950868 [ 5790.160948] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5790.175130] LustreError: 178976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5790.535585] Lustre: server umount lustre-MDT0001 complete [ 5802.733175] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 5805.543945] Key type lgssc unregistered [ 5805.878725] LNet: 182818:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5805.887038] LNetError: 182818:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5805.905222] LNet: Removed LNI 192.168.201.133@tcp [ 5806.784925] Key type .llcrypt unregistered [ 5806.788310] Key type ._llcrypt unregistered [ 5833.908675] Key type ._llcrypt registered [ 5833.911655] Key type .llcrypt registered [ 5834.105119] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 5835.715515] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5835.906271] alg: No test for adler32 (adler32-zlib) [ 5836.942680] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 5837.567637] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 5839.408414] Key type lgssc registered [ 5841.359000] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5854.169691] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5854.199312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5856.254177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5861.350822] LustreError: 184229:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5861.385358] LustreError: 184229:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5863.038986] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 5866.479173] LustreError: 184230:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5871.591057] LustreError: 184229:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5873.657127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5874.448917] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5880.215754] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 5888.881292] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5896.897304] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5905.912811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5906.256191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5913.336048] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 5922.063885] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5926.775884] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5950.334633] Lustre: server umount lustre-OST0000 complete [ 5956.578536] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5956.584654] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5956.601536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5957.605220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5957.607197] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5962.722489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5962.728586] Lustre: Skipped 2 previous similar messages [ 5964.870723] Lustre: server umount lustre-MDT0000 complete [ 5965.791389] Lustre: 183538:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788442897/real 1788442897] req@00000000ba331e58 x1875318180710400/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788442913 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5965.849369] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5965.871089] LustreError: 184226:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5965.903782] LustreError: 184226:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5967.845580] Lustre: 183539:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788442899/real 1788442899] req@000000005841a2f0 x1875318180711552/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788442915 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5969.163250] LustreError: 184212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788442916 with bad export cookie 12407966309674275810 [ 5969.176092] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5969.393194] Lustre: server umount lustre-MDT0001 complete [ 5979.668363] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 5982.469961] Key type lgssc unregistered [ 5982.787221] LNet: 187788:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5982.797044] LNetError: 187788:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5982.817348] LNet: Removed LNI 192.168.201.133@tcp [ 5983.701763] Key type .llcrypt unregistered [ 5983.703893] Key type ._llcrypt unregistered [ 6011.051675] Key type ._llcrypt registered [ 6011.055154] Key type .llcrypt registered [ 6011.229571] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 6012.989979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6013.067535] alg: No test for adler32 (adler32-zlib) [ 6014.349801] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 6014.737090] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 6016.543198] Key type lgssc registered [ 6018.388773] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6027.771799] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6027.803844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6029.199117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6034.755641] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6039.520237] LustreError: 189204:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6039.542222] LustreError: 189204:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6044.641660] LustreError: 189205:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6046.363990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6047.165321] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6052.323670] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6063.441396] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6070.285414] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6079.117974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6079.655221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6088.963537] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6098.275676] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6099.121321] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6106.441588] Lustre: lustre-OST0000: Not available for connect from 192.168.201.33@tcp (stopping) [ 6107.616712] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6107.624633] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6107.639319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6109.666871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6109.668514] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6109.695365] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6114.785787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6114.818124] Lustre: Skipped 3 previous similar messages [ 6119.391602] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6119.624536] Lustre: server umount lustre-OST0000 complete [ 6129.010396] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff939cd00d1800.ost_server_uuid 50 [ 6131.673960] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6149.603050] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6149.615206] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6149.637095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6149.648754] Lustre: Skipped 1 previous similar message [ 6152.295683] Lustre: server umount lustre-MDT0000 complete [ 6155.744670] LustreError: 189200:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6155.756974] LustreError: 189200:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6156.064432] LustreError: 192251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443103 with bad export cookie 8881396688400094399 [ 6156.069960] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6156.073945] LustreError: 192251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6156.423644] Lustre: server umount lustre-MDT0001 complete [ 6166.542790] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 09:45:11 (1788443111) [ 6175.244184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6176.070277] LustreError: 193122:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6176.163894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6181.345346] LustreError: 193123:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6181.640911] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6191.585746] LustreError: 193122:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6191.609715] LustreError: 193122:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6191.711225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6192.303544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6198.314823] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6208.254154] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6215.485897] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6224.172139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6224.724091] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6232.277847] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6242.011714] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6246.278178] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6252.399400] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6252.743696] Lustre: lustre-OST0001: new disk, initializing [ 6252.750353] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6252.760550] Lustre: 195966:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6252.964459] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6259.967941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6259.973217] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6260.131873] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6261.266653] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6274.156608] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6278.112581] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6278.135310] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6278.153902] Lustre: Skipped 1 previous similar message [ 6278.162257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6278.175491] Lustre: Skipped 2 previous similar messages [ 6284.442662] Lustre: server umount lustre-OST0000 complete [ 6285.117682] LustreError: 194945:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6292.430938] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6292.720357] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6299.240428] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6299.473445] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6316.518211] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6316.520659] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6316.520711] Lustre: Skipped 1 previous similar message [ 6316.521595] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6316.530524] Lustre: Skipped 3 previous similar messages [ 6316.573991] LustreError: Skipped 1 previous similar message [ 6320.263549] Lustre: server umount lustre-OST0001 complete [ 6325.217894] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6325.231310] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6325.243531] Lustre: Skipped 1 previous similar message [ 6330.490441] Lustre: server umount lustre-MDT0000 complete [ 6331.880131] LustreError: 193118:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6331.891346] LustreError: 193118:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [ 6334.912467] LustreError: 197536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443282 with bad export cookie 8881396688400095470 [ 6334.919171] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6334.932428] LustreError: 197536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6335.290647] Lustre: server umount lustre-MDT0001 complete [ 6360.806950] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 09:48:26 (1788443306) [ 6368.155385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6368.396326] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6368.420317] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6368.600702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6373.396768] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6383.172913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6383.319050] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6383.381643] Lustre: 199406:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6389.037872] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6397.149275] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6403.606515] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6411.492256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6411.708282] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6417.904772] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6421.675758] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6426.354404] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6436.465124] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6436.646348] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6437.070185] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6437.076858] Lustre: Skipped 2 previous similar messages [ 6444.746616] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6444.889374] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6455.520306] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6460.422316] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6460.432526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6460.442074] Lustre: Skipped 2 previous similar messages [ 6460.462239] Lustre: Skipped 5 previous similar messages [ 6466.047101] Lustre: server umount lustre-OST0000 complete [ 6466.873051] LustreError: 201260:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6486.016205] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6486.032470] LustreError: Skipped 1 previous similar message [ 6490.770066] Lustre: server umount lustre-OST0001 complete [ 6496.225622] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6496.239262] Lustre: Skipped 3 previous similar messages [ 6510.047167] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6510.316699] Lustre: server umount lustre-MDT0000 complete [ 6514.881078] LustreError: 199399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443462 with bad export cookie 8881396688400096877 [ 6514.883316] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6514.893066] LustreError: 199399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6515.295972] Lustre: server umount lustre-MDT0001 complete [ 6544.806880] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 09:51:29 (1788443489) [ 6566.297637] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 6577.512240] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 6619.850247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6620.141141] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6620.168640] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6620.220228] Lustre: lustre-MDT0000: new disk, initializing [ 6620.274774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6620.286359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6624.854728] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6636.471367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6636.568458] Lustre: 207514:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6636.625658] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6636.628713] Lustre: Skipped 1 previous similar message [ 6636.703651] Lustre: lustre-MDT0001: new disk, initializing [ 6636.823695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6636.834933] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6640.515560] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6652.164340] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6659.924505] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6668.474255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6668.770329] Lustre: lustre-OST0000: new disk, initializing [ 6668.782168] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6668.796356] Lustre: 209479:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6677.077509] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6678.676363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6678.693512] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6678.884804] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6689.039504] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6695.492866] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6695.850464] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6701.965886] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6702.211581] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6704.608073] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6704.615260] LustreError: Skipped 1 previous similar message [ 6704.616853] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6704.622979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6704.622987] Lustre: Skipped 11 previous similar messages [ 6704.635266] Lustre: Skipped 1 previous similar message [ 6710.885608] Lustre: server umount lustre-OST0000 complete [ 6721.875983] Lustre: server umount lustre-MDT0000 complete [ 6725.093420] LustreError: 207523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6725.117663] LustreError: 207523:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 6726.288105] LustreError: 207505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443673 with bad export cookie 8881396688400098662 [ 6726.310586] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6726.993272] Lustre: server umount lustre-MDT0001 complete [ 6737.676869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6743.495512] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6753.158819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6760.230548] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6769.539309] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6775.229408] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6782.462831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6789.879735] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6798.172571] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6803.097394] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6803.472745] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6809.610306] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6809.845610] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6813.680338] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6813.688381] LustreError: Skipped 1 previous similar message [ 6818.151408] Lustre: server umount lustre-OST0000 complete [ 6825.709708] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6831.928837] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6861.422128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6868.384551] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6876.258021] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6879.077700] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 6879.145963] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6879.175598] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6879.177261] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6879.214584] Lustre: Skipped 1 previous similar message [ 6884.324523] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6884.344035] Lustre: Skipped 5 previous similar messages [ 6884.347439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6884.365986] Lustre: Skipped 9 previous similar messages [ 6889.881141] Lustre: server umount lustre-OST0000 complete [ 6908.383369] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6908.727596] Lustre: server umount lustre-MDT0000 complete [ 6912.460887] LustreError: 212431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788443859 with bad export cookie 8881396688400099544 [ 6912.467478] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6912.473737] LustreError: 212431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6912.942915] Lustre: server umount lustre-MDT0001 complete [ 6922.629912] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 09:57:47 (1788443867) [ 6930.158792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6930.803448] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6930.809221] Lustre: Skipped 6 previous similar messages [ 6935.223211] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6944.778660] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6950.163565] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6958.042486] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6963.398694] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6971.250772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6978.722673] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 6987.804946] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6991.688403] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6991.832601] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6996.024915] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6996.351792] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7006.200395] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7006.478656] Lustre: lustre-OST0001: new disk, initializing [ 7006.482137] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7006.498207] Lustre: 221037:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7015.736331] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7016.627221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7016.650635] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7016.788777] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7027.704087] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7033.262543] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7033.467157] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7037.930971] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7038.130121] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7042.538060] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7042.575664] LustreError: Skipped 3 previous similar messages [ 7046.585879] Lustre: server umount lustre-OST0001 complete [ 7047.672664] LustreError: 219489:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7047.700958] LustreError: 219489:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 7054.381109] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7054.852509] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7062.189860] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7062.677117] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7092.097670] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7097.837615] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7097.861213] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7097.866960] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7097.908967] Lustre: Skipped 1 previous similar message [ 7099.398261] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7111.716078] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7115.211619] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7120.318969] Lustre: server umount lustre-OST0001 complete [ 7143.907627] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7143.934844] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7143.958104] Lustre: Skipped 7 previous similar messages [ 7143.971523] Lustre: Skipped 12 previous similar messages [ 7147.188802] Lustre: server umount lustre-OST0000 complete [ 7158.336495] Lustre: server umount lustre-MDT0000 complete [ 7162.980945] LustreError: 217646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444110 with bad export cookie 8881396688400100510 [ 7162.990579] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7163.004741] LustreError: 217646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7169.560718] Lustre: server umount lustre-MDT0001 complete [ 7200.375464] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 10:02:25 (1788444145) [ 7206.191150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7206.492285] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7206.560841] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7210.373452] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7220.212182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7220.426206] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7220.472600] Lustre: 226292:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7225.005316] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7232.962489] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7240.892287] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7250.276641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7250.618626] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7257.542667] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7266.117890] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7277.878809] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7277.984962] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7285.438230] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7295.866994] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7298.112791] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7304.735166] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7304.977906] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7311.153810] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7311.354846] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7317.168875] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff939cd00d0800.ost_server_uuid 50 [ 7319.184865] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff939cd00d0800.ost_server_uuid in IDLE state after 0 sec [ 7320.442815] Lustre: Permanently deactivating lustre-OST0001 [ 7320.469340] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7323.277171] Lustre: Permanently reactivating lustre-OST0001 [ 7327.712936] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7327.726120] LustreError: Skipped 3 previous similar messages [ 7332.238185] Lustre: server umount lustre-OST0001 complete [ 7350.240503] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7350.561840] Lustre: server umount lustre-OST0000 complete [ 7372.259598] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7372.732879] Lustre: server umount lustre-MDT0000 complete [ 7377.005872] LustreError: 226284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444324 with bad export cookie 8881396688400101798 [ 7377.014522] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7377.019681] LustreError: 226284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7377.447779] Lustre: server umount lustre-MDT0001 complete [ 7403.995936] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 10:05:49 (1788444349) [ 7412.099605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7412.453548] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7412.512558] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7412.523190] Lustre: Skipped 5 previous similar messages [ 7419.131700] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7428.240102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7428.427583] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7428.493338] Lustre: 232994:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7433.719615] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7441.764875] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7447.802320] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7454.706942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7454.963952] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7454.982761] Lustre: Permanently deactivating lustre-OST0000 [ 7454.995525] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7455.270192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7455.279641] Lustre: Skipped 10 previous similar messages [ 7462.271839] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7464.987485] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7470.781548] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7478.773940] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7478.860134] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7487.365089] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7488.515155] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7497.418587] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7499.736080] Lustre: Permanently reactivating lustre-OST0000 [ 7499.775138] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7499.778671] Lustre: Skipped 2 previous similar messages [ 7505.468807] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7505.469439] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 40 seconds. I think it's dead, and I am evicting it. exp 0000000002a66b46, cur 1788444452 deadline 1788444442 last 1788444412 [ 7505.474660] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7505.518982] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7505.643923] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7523.137508] Lustre: server umount lustre-OST0001 complete [ 7533.634689] Lustre: server umount lustre-OST0000 complete [ 7544.160242] Lustre: server umount lustre-MDT0000 complete [ 7548.488399] LustreError: 232985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444495 with bad export cookie 8881396688400103443 [ 7548.494696] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7548.497613] LustreError: 232985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7548.812236] Lustre: server umount lustre-MDT0001 complete [ 7561.460750] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 7564.321484] Key type lgssc unregistered [ 7564.586979] LNet: 237876:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7564.593509] LNetError: 237876:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7564.607755] LNet: Removed LNI 192.168.201.133@tcp [ 7565.571684] Key type .llcrypt unregistered [ 7565.574169] Key type ._llcrypt unregistered [ 7589.166113] Key type ._llcrypt registered [ 7589.173246] Key type .llcrypt registered [ 7589.368170] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 7590.810555] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7591.037326] alg: No test for adler32 (adler32-zlib) [ 7592.505844] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 7592.832800] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 7594.607181] Key type lgssc registered [ 7596.242961] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7607.829450] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7607.849033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7609.492146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7615.469801] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7619.043957] LustreError: 239338:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7619.088655] LustreError: 239338:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7624.161504] LustreError: 239339:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7625.520480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7625.599982] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7625.625880] Lustre: 239326:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7625.640608] Lustre: Permanently deactivating lustre-MDT0001 [ 7625.655849] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7625.897100] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7627.499098] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7627.519223] LustreError: 239370:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7627.535857] LustreError: 239370:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7627.544774] LustreError: 239370:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 1, failed: rc = -108 [ 7630.823146] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7639.459465] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7645.652996] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7653.446162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7653.647455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7654.716970] LustreError: 241185:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7654.747580] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7660.011874] LustreError: 241185:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7660.199537] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7665.127845] LustreError: 241184:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7669.527983] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7671.289575] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7676.388217] LustreError: 241182:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7676.423480] LustreError: 241182:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 7677.841427] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7678.169330] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7683.602505] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7683.604062] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7686.030458] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7693.808773] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7697.202417] Lustre: Permanently reactivating lustre-MDT0001 [ 7697.215428] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7697.219459] Lustre: Skipped 2 previous similar messages [ 7702.885592] Lustre: lustre-MDT0001: Client d9e7ebc2-a123-4527-bdad-51c3d79036b1 (at 192.168.201.33@tcp) reconnecting [ 7710.246468] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7711.727970] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7717.056854] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7717.256478] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7722.534957] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7722.972722] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7725.878235] Lustre: Permanently deactivating lustre-MDT0001 [ 7725.890888] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7725.894840] Lustre: Skipped 2 previous similar messages [ 7731.799572] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7747.042063] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7747.048192] Lustre: Skipped 2 previous similar messages [ 7747.050428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7747.059558] Lustre: Skipped 3 previous similar messages [ 7750.357071] Lustre: server umount lustre-MDT0000 complete [ 7752.184235] LustreError: 239339:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7752.198918] LustreError: 239339:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7754.515885] LustreError: 239318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444701 with bad export cookie 487128633938828796 [ 7754.531309] LustreError: 239318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7754.536915] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7755.114071] Lustre: server umount lustre-MDT0001 complete [ 7760.225570] Lustre: server umount lustre-OST0000 complete [ 7774.342734] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 10:11:59 (1788444719) [ 7789.586795] Lustre: server umount lustre-OST0001 complete [ 7797.811213] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 7810.070883] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 7855.459140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7855.635681] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7855.641109] Lustre: Skipped 2 previous similar messages [ 7855.655786] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7855.724485] Lustre: lustre-MDT0000: new disk, initializing [ 7855.772147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7855.786685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7860.269359] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7872.922078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7873.028362] Lustre: 247624:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7873.136512] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7873.144653] Lustre: Skipped 1 previous similar message [ 7873.284449] Lustre: lustre-MDT0001: new disk, initializing [ 7873.466526] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7873.557545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7873.594576] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7879.369871] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7890.314896] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7896.222523] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7903.455908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7904.023997] Lustre: lustre-OST0000: new disk, initializing [ 7904.028048] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7904.034634] Lustre: 249591:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7904.216028] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7911.216875] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7912.016080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7912.030695] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7912.169965] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7924.143141] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7927.533975] LustreError: 247631:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7937.407711] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7937.551165] LustreError: 247631:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7937.579187] LustreError: 248993:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7937.702055] Lustre: lustre-OST0001: new disk, initializing [ 7937.709859] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7937.716543] Lustre: 250706:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7937.922185] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7945.877158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7945.893043] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7946.009573] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 7946.046731] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7947.624133] LustreError: 248993:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7957.351451] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7959.415236] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7966.723375] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7966.734641] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7966.752502] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7966.780259] Lustre: Skipped 1 previous similar message [ 7968.894720] Lustre: server umount lustre-OST0001 complete [ 7971.829111] LustreError: 251069:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7971.838397] LustreError: 251069:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7989.216839] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7989.218032] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7989.235926] LustreError: Skipped 2 previous similar messages [ 7989.237655] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7989.267259] Lustre: Skipped 2 previous similar messages [ 7992.295432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7992.303026] Lustre: Skipped 2 previous similar messages [ 7994.257391] Lustre: server umount lustre-OST0000 complete [ 8001.506914] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8001.520382] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8001.571044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8012.769774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8012.781780] Lustre: Skipped 5 previous similar messages [ 8013.281368] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8015.585933] Lustre: server umount lustre-MDT0000 complete [ 8020.410641] LustreError: 250705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788444967 with bad export cookie 487128633938831603 [ 8020.420164] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8020.422843] LustreError: 250705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8027.077198] Lustre: server umount lustre-MDT0001 complete [ 8041.474710] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 8044.443639] Key type lgssc unregistered [ 8044.751241] LNet: 252744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8044.777369] LNetError: 252744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8044.801895] LNet: Removed LNI 192.168.201.133@tcp [ 8045.768285] Key type .llcrypt unregistered [ 8045.771284] Key type ._llcrypt unregistered [ 8095.670848] Key type ._llcrypt registered [ 8095.678735] Key type .llcrypt registered [ 8095.883949] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 8098.560466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8098.633902] alg: No test for adler32 (adler32-zlib) [ 8099.868592] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8100.325533] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 8102.071582] Key type lgssc registered [ 8103.447356] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8115.902135] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8115.915394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8117.284807] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8117.344600] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8117.485714] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8123.248725] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 8133.292205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8133.432346] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8133.501695] Lustre: 254952:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8133.931778] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8139.880968] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 8148.854846] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8156.241976] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8164.575231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8164.821575] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8165.107823] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8170.739770] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8172.605266] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 8181.958559] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8237.540001] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8237.547754] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8237.553435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8237.566539] Lustre: Skipped 1 previous similar message [ 8237.575071] Lustre: Skipped 1 previous similar message [ 8242.670761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8242.690953] Lustre: Skipped 1 previous similar message [ 8243.106641] Lustre: server umount lustre-OST0000 complete [ 8248.139262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8263.323136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8263.687834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8268.778057] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8268.806220] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8268.808528] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8268.830062] Lustre: Skipped 1 previous similar message [ 8271.152320] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 8279.371446] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8325.090718] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8325.097311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8325.097947] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8325.097952] LustreError: Skipped 1 previous similar message [ 8325.101249] Lustre: Skipped 1 previous similar message [ 8325.127536] Lustre: Skipped 1 previous similar message [ 8326.892080] Lustre: server umount lustre-OST0000 complete [ 8333.801471] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8333.804178] LustreError: Skipped 1 previous similar message [ 8333.811746] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8333.827132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8337.899708] Lustre: server umount lustre-MDT0000 complete [ 8342.647047] LustreError: 254945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445289 with bad export cookie 18422361206856106216 [ 8342.652324] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8342.674810] LustreError: 254945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8342.922915] Lustre: server umount lustre-MDT0001 complete [ 8359.800924] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 8363.926711] Key type lgssc unregistered [ 8364.415275] LNet: 260379:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8364.426563] LNetError: 260379:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8364.452986] LNet: Removed LNI 192.168.201.133@tcp [ 8365.711962] Key type .llcrypt unregistered [ 8365.720976] Key type ._llcrypt unregistered [ 8392.660694] Key type ._llcrypt registered [ 8392.663398] Key type .llcrypt registered [ 8392.761836] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 8394.020275] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8394.198290] alg: No test for adler32 (adler32-zlib) [ 8395.235559] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8395.642752] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 8397.387599] Key type lgssc registered [ 8398.267163] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8405.073334] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8405.095269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8406.402791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8411.392421] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 8416.737153] LustreError: 261794:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8416.756458] LustreError: 261794:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8421.385383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8421.998176] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8427.003436] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 8437.397276] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8446.745659] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8455.844834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8456.384267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8464.313299] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 8474.918624] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8475.879581] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8506.854205] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8506.857106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8506.866852] Lustre: Skipped 1 previous similar message [ 8506.869055] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8506.897204] Lustre: Skipped 1 previous similar message [ 8511.389869] Lustre: server umount lustre-OST0000 complete [ 8516.781549] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8516.807455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8522.210182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8522.219654] Lustre: Skipped 2 previous similar messages [ 8523.532341] Lustre: server umount lustre-MDT0000 complete [ 8527.329863] LustreError: 261794:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8527.360654] LustreError: 261794:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8527.467211] LustreError: 261775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445474 with bad export cookie 6160020419650877402 [ 8527.483194] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8528.076994] Lustre: server umount lustre-MDT0001 complete [ 8538.624652] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 8541.880551] Key type lgssc unregistered [ 8542.257255] LNet: 266103:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8542.262216] LNetError: 266103:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8543.269928] LNet: Removed LNI 192.168.201.133@tcp [ 8544.150294] Key type .llcrypt unregistered [ 8544.156045] Key type ._llcrypt unregistered [ 8562.358208] Key type ._llcrypt registered [ 8562.360344] Key type .llcrypt registered [ 8562.501417] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 8563.832469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8563.918074] alg: No test for adler32 (adler32-zlib) [ 8565.135942] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8565.397443] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 8567.087877] Key type lgssc registered [ 8568.357296] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8576.376032] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8576.420468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8578.055868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8582.232113] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 8588.262672] LustreError: 267321:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8588.277819] LustreError: 267321:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8590.466911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8590.753670] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8595.967912] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 8603.741080] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8610.108691] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8616.288990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8616.590063] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8621.785740] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 8623.800096] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8632.445580] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8649.709442] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8649.726519] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8649.760759] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8650.724257] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8650.730174] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8650.742587] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8655.816861] Lustre: server umount lustre-OST0000 complete [ 8662.500130] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8662.508348] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8662.514911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8662.517536] Lustre: Skipped 1 previous similar message [ 8666.082023] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8666.085067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8666.124682] Lustre: Skipped 1 previous similar message [ 8668.303092] Lustre: server umount lustre-MDT0000 complete [ 8671.203748] LustreError: 268753:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8671.216649] LustreError: 268753:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8672.142631] LustreError: 267301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445619 with bad export cookie 7264614741111507026 [ 8672.149343] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8672.154107] LustreError: 267301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8672.635947] Lustre: server umount lustre-MDT0001 complete [ 8684.937374] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 8688.026532] Key type lgssc unregistered [ 8688.302740] LNet: 270999:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8688.309150] LNetError: 270999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8688.328228] LNet: Removed LNI 192.168.201.133@tcp [ 8689.216954] Key type .llcrypt unregistered [ 8689.218490] Key type ._llcrypt unregistered [ 8717.789400] Key type ._llcrypt registered [ 8717.791750] Key type .llcrypt registered [ 8717.915759] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 8719.218217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8719.403509] alg: No test for adler32 (adler32-zlib) [ 8720.470830] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8720.770703] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 8722.535167] Key type lgssc registered [ 8723.923589] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8732.165693] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8732.219170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8733.697842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8739.276712] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 8742.884974] LustreError: 272415:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8742.904739] LustreError: 272415:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8748.002316] LustreError: 272416:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8749.577800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8749.965353] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8755.659602] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 8765.388797] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8773.290486] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8780.681611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8780.930969] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8786.491107] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 8796.758450] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8803.517359] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8831.470048] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8831.479703] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8831.493373] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8834.539409] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8834.540245] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8834.547379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8837.683229] Lustre: server umount lustre-OST0000 complete [ 8842.209388] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8842.227000] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8842.253256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8842.255784] Lustre: Skipped 1 previous similar message [ 8844.767978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8844.769833] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8848.427601] Lustre: server umount lustre-MDT0000 complete [ 8849.892737] LustreError: 272410:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8849.919776] LustreError: 272410:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8852.663335] LustreError: 272395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445799 with bad export cookie 12844905605486065435 [ 8852.666641] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8852.674910] LustreError: 272395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8853.051849] Lustre: server umount lustre-MDT0001 complete [ 8865.062770] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 8868.303694] Key type lgssc unregistered [ 8868.693862] LNet: 276744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8868.711787] LNetError: 276744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8869.745135] LNet: Removed LNI 192.168.201.133@tcp [ 8870.652525] Key type .llcrypt unregistered [ 8870.654922] Key type ._llcrypt unregistered [ 8891.178789] Key type ._llcrypt registered [ 8891.181671] Key type .llcrypt registered [ 8891.354734] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 8892.663762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8892.853987] alg: No test for adler32 (adler32-zlib) [ 8894.051964] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 8894.310559] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 8895.983212] Key type lgssc registered [ 8897.179727] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8905.592587] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8905.606214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8906.996850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8912.456701] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 8917.478849] LustreError: 277978:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8917.491176] LustreError: 277978:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8922.098719] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8922.489462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8928.258348] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 8936.683603] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8943.287669] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8951.009874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8951.340163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8952.379114] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8957.203657] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 8967.723924] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8983.010622] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8983.011951] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8983.016341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8983.022866] LustreError: Skipped 1 previous similar message [ 8983.041973] Lustre: Skipped 1 previous similar message [ 8988.148098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8988.160507] Lustre: Skipped 1 previous similar message [ 8993.249647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8993.261353] Lustre: Skipped 2 previous similar messages [ 8995.295314] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8995.666748] Lustre: server umount lustre-OST0000 complete [ 9003.490891] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9003.515294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9006.377648] Lustre: server umount lustre-MDT0000 complete [ 9008.612295] LustreError: 277962:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9008.633239] LustreError: 277962:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9010.724693] LustreError: 277942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788445957 with bad export cookie 9845636962184614642 [ 9010.740296] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9010.769775] LustreError: 277942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9011.085524] Lustre: server umount lustre-MDT0001 complete [ 9021.777704] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 9025.286506] Key type lgssc unregistered [ 9025.713875] LNet: 281680:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9025.732192] LNetError: 281680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9025.768489] LNet: Removed LNI 192.168.201.133@tcp [ 9026.913043] Key type .llcrypt unregistered [ 9026.923095] Key type ._llcrypt unregistered [ 9041.068230] Key type ._llcrypt registered [ 9041.070418] Key type .llcrypt registered [ 9041.251486] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9063.196322] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 9081.548684] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 9083.158691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9083.227715] alg: No test for adler32 (adler32-zlib) [ 9084.470766] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 9084.907667] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 9086.689725] Key type lgssc registered [ 9087.959862] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9131.802500] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9131.836948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9133.163288] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9133.199991] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9133.301452] Lustre: lustre-MDT0000: new disk, initializing [ 9133.393851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9133.452979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9139.985296] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 9151.870312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9152.051710] Lustre: 285235:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9152.140932] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9152.143198] Lustre: Skipped 1 previous similar message [ 9152.224170] Lustre: lustre-MDT0001: new disk, initializing [ 9152.353826] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9152.412943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9152.429756] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9157.870548] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 9168.021374] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9173.718048] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9181.229607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9181.653982] Lustre: lustre-OST0000: new disk, initializing [ 9181.657328] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9181.662652] Lustre: 287208:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9181.776083] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9189.081804] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 9189.464878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9189.475445] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9189.643016] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9199.230790] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9204.750876] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9205.000536] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9210.830393] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9211.134450] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9215.464056] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9215.486111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9215.503020] Lustre: Skipped 1 previous similar message [ 9215.512566] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9219.426060] Lustre: server umount lustre-OST0000 complete [ 9224.160531] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9224.165746] LustreError: Skipped 1 previous similar message [ 9224.170462] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9224.190330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9224.196235] Lustre: Skipped 1 previous similar message [ 9225.696820] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9225.698810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9225.720765] Lustre: Skipped 1 previous similar message [ 9229.845381] Lustre: server umount lustre-MDT0000 complete [ 9234.674725] LustreError: 285229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446181 with bad export cookie 6994880274089388552 [ 9234.675351] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9234.686516] LustreError: 285229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9235.050487] Lustre: server umount lustre-MDT0001 complete [ 9245.817876] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 10:36:30 (1788446190) [ 9254.786397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9255.913249] LustreError: 289619:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9255.929245] LustreError: 289619:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9255.994648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9261.040386] LustreError: 289620:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9262.216681] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 9266.146234] LustreError: 289619:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9271.280478] LustreError: 289620:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9272.111887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9272.601119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9277.868288] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 9286.096532] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9291.518934] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9297.885909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9298.223063] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9305.931157] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 9313.404773] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9328.101182] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9328.122191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9328.124116] Lustre: Skipped 1 previous similar message [ 9328.128598] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9333.228423] LustreError: 291442:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9333.371194] Lustre: server umount lustre-OST0000 complete [ 9338.352364] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9338.353879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9338.374947] Lustre: Skipped 1 previous similar message [ 9338.394842] Lustre: Skipped 2 previous similar messages [ 9343.720777] Lustre: server umount lustre-MDT0000 complete [ 9347.490917] LustreError: 289600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446294 with bad export cookie 6994880274089389434 [ 9347.494757] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9347.505455] LustreError: 289600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9347.970713] Lustre: server umount lustre-MDT0001 complete [ 9360.936746] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 9363.407721] Key type lgssc unregistered [ 9363.787712] LNet: 293171:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9363.807231] LNetError: 293171:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9363.827286] LNet: Removed LNI 192.168.201.133@tcp [ 9364.602273] Key type .llcrypt unregistered [ 9364.609682] Key type ._llcrypt unregistered [ 9393.551899] Key type ._llcrypt registered [ 9393.555377] Key type .llcrypt registered [ 9393.744558] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 9409.414341] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 9411.053971] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9411.228210] alg: No test for adler32 (adler32-zlib) [ 9412.382771] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 9412.754471] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 9414.463188] Key type lgssc registered [ 9415.848163] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9462.219552] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9462.259839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9463.645563] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9463.668878] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9463.784455] Lustre: lustre-MDT0000: new disk, initializing [ 9463.889132] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9463.922988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9468.879543] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 9480.536151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9480.681661] Lustre: 297000:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9480.785581] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9480.790215] Lustre: Skipped 1 previous similar message [ 9480.937523] Lustre: lustre-MDT0001: new disk, initializing [ 9481.069398] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9481.131708] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9481.151200] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9486.732887] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 9496.435582] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9502.099282] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9510.115992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9510.531341] Lustre: lustre-OST2710: new disk, initializing [ 9510.537918] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9510.544189] Lustre: 298967:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9510.631897] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9516.878671] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 9517.157134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9517.170110] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9517.259466] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9526.206379] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9533.232746] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9533.396973] Lustre: lustre-OST03e8: new disk, initializing [ 9533.403918] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9533.408362] Lustre: 300032:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9533.515556] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9539.179208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9539.189951] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9539.327734] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9540.389871] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 9548.635041] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9557.352433] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9557.599912] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9562.367977] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9562.564548] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9567.715646] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9567.955997] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9573.107690] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9573.431445] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9578.465639] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9578.477351] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9578.495667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9579.861643] Lustre: server umount lustre-MDT0000 complete [ 9580.514570] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9583.938851] LustreError: 296992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446531 with bad export cookie 8055722749970845424 [ 9583.945940] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9583.952977] LustreError: 296992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9584.477460] Lustre: server umount lustre-MDT0001 complete [ 9599.437564] Lustre: server umount lustre-OST2710 complete [ 9601.507111] Lustre: 294793:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788446532/real 1788446532] req@00000000fdc07abc x1875321929405312/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788446548 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9601.530578] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9601.540979] Lustre: Skipped 2 previous similar messages [ 9603.502914] Lustre: server umount lustre-OST03e8 complete [ 9624.373454] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [ 9633.369745] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 9671.636903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9671.912468] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9671.938340] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9672.007631] Lustre: lustre-MDT0000: new disk, initializing [ 9672.050583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9672.084156] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9676.347172] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 9687.086403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9687.277394] Lustre: 305277:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9687.360140] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9687.367093] Lustre: Skipped 1 previous similar message [ 9687.464375] Lustre: lustre-MDT0001: new disk, initializing [ 9687.621048] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9687.683645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9687.694863] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9693.170838] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 9703.759548] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9710.957842] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9720.109482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9720.429803] Lustre: lustre-OST0000: new disk, initializing [ 9720.435718] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9720.444670] Lustre: 307247:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9720.606558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9728.739951] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 9729.125376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9729.144353] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9729.300155] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9737.967472] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9744.151217] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9744.411525] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9749.129327] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9749.291375] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9755.106469] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9755.108462] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9755.116826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9755.140603] Lustre: Skipped 1 previous similar message [ 9758.085627] Lustre: server umount lustre-OST0000 complete [ 9764.334888] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9764.337780] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9764.363310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9764.370717] Lustre: Skipped 1 previous similar message [ 9770.465619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9770.468089] Lustre: Skipped 2 previous similar messages [ 9775.585265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9775.590837] Lustre: Skipped 1 previous similar message [ 9777.119155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9777.289118] Lustre: server umount lustre-MDT0000 complete [ 9780.704744] LustreError: 307461:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9780.719752] LustreError: 307461:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 9781.179195] LustreError: 305270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446728 with bad export cookie 8055722749970847097 [ 9781.186335] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9781.194150] LustreError: 305270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9781.591416] Lustre: server umount lustre-MDT0001 complete [ 9789.779272] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 10:45:35 (1788446735) [ 9791.370717] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9793.441258] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 10:45:38 (1788446738) [ 9797.522266] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 9825.581883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9825.828471] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9825.871726] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9826.123094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9830.192056] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 9838.674752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9838.832136] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9838.889851] Lustre: 311109:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9844.708945] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 9867.834341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9868.176826] Lustre: Denying initial registration attempt from nid 192.168.201.133@tcp, specified as failover [ 9868.188233] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9868.205603] LustreError: 312563:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -99 [ 9868.212440] LustreError: 312563:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9868.218306] LustreError: 312563:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9868.303461] Lustre: server umount lustre-OST0000 complete [ 9868.306699] LustreError: 312563:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9876.963506] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9876.978581] Lustre: Skipped 1 previous similar message [ 9876.986170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9876.994941] Lustre: Skipped 1 previous similar message [ 9882.088145] LustreError: 311121:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9882.101870] LustreError: 311121:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9882.161585] Lustre: server umount lustre-MDT0000 complete [ 9886.300964] LustreError: 311102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788446833 with bad export cookie 8055722749970847986 [ 9886.309519] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9886.315443] LustreError: 311102:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9886.722137] Lustre: server umount lustre-MDT0001 complete [ 9898.447127] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [ 9901.740822] Key type lgssc unregistered [ 9902.047813] LNet: 313682:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9902.059116] LNetError: 313682:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9902.080422] LNet: Removed LNI 192.168.201.133@tcp [ 9902.854989] Key type .llcrypt unregistered [ 9902.856723] Key type ._llcrypt unregistered [ 9914.669171] Key type ._llcrypt registered [ 9914.674328] Key type .llcrypt registered [ 9914.811221] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 9915.754261] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9915.932088] alg: No test for adler32 (adler32-zlib) [ 9916.927241] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [ 9917.112331] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [ 9918.767205] Key type lgssc registered [ 9919.855535] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9965.388678] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [ 9974.088415] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9974.103142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9975.341094] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9975.373691] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9975.485524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9980.335310] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [ 9990.174567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9990.266602] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9990.282201] Lustre: 316446:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9990.557707] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9995.642243] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10003.352611] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10009.424644] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10030.125829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10030.376935] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10030.612488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10037.500394] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10044.716727] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10050.528253] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10050.534682] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10050.543959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10051.042552] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10051.058302] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10051.061063] Lustre: Skipped 1 previous similar message [10056.167564] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10056.170944] Lustre: Skipped 1 previous similar message [10061.283970] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10061.287096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10061.330426] Lustre: Skipped 1 previous similar message [10061.530600] Lustre: server umount lustre-OST0000 complete [10066.403524] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10066.411198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10066.432173] Lustre: Skipped 1 previous similar message [10071.541992] Lustre: server umount lustre-MDT0000 complete [10074.980508] LustreError: 316435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447022 with bad export cookie 16599145339292345859 [10074.985599] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10074.986852] LustreError: 316435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10075.218374] Lustre: server umount lustre-MDT0001 complete [10085.752203] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [10088.140785] Key type lgssc unregistered [10088.476167] LNet: 320021:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10088.492650] LNetError: 320021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10088.514300] LNet: Removed LNI 192.168.201.133@tcp [10089.388832] Key type .llcrypt unregistered [10089.392796] Key type ._llcrypt unregistered [10113.226439] Key type ._llcrypt registered [10113.231865] Key type .llcrypt registered [10113.347863] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [10114.263852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10114.354337] alg: No test for adler32 (adler32-zlib) [10115.623977] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [10115.894738] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [10117.631239] Key type lgssc registered [10118.811530] Lustre: Echo OBD driver; http://www.lustre.org/ [10126.042619] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10126.057239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10127.412777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10130.612281] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10137.575304] LustreError: 321433:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10137.591032] LustreError: 321433:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10139.920617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10140.411514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10146.056727] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10154.831556] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10161.380841] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10170.313294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10170.886568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10177.684296] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10186.048978] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10195.427057] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10195.428776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10195.432185] Lustre: Skipped 2 previous similar messages [10195.437552] Lustre: Skipped 1 previous similar message [10198.386513] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.33@tcp (stopping) [10198.397277] Lustre: Skipped 1 previous similar message [10200.265409] Lustre: server umount lustre-MDT0000 complete [10200.555029] LustreError: 321427:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10200.577918] LustreError: 321427:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10203.479367] LustreError: 322554:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10204.699382] LustreError: 321412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447151 with bad export cookie 14591696193577215536 [10204.712526] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10204.727240] LustreError: 321412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10205.664455] LustreError: 321427:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10205.665315] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10205.699245] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10208.617899] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.33@tcp (stopping) [10209.697170] LustreError: 321429:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10209.710673] LustreError: 321429:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10211.236147] Lustre: server umount lustre-MDT0001 complete [10218.714527] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10229.028272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10249.960552] LustreError: 323265:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10250.024486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10250.070370] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10254.526194] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10262.461928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10262.725763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10263.788162] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10267.344909] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10274.735574] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10276.014087] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10282.633963] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10284.572850] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10291.177543] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10291.187350] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10291.201269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10291.205628] Lustre: Skipped 1 previous similar message [10293.729509] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10293.732870] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10295.865310] Lustre: server umount lustre-OST0000 complete [10299.874076] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10299.884126] Lustre: Skipped 1 previous similar message [10299.888036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10299.891826] Lustre: Skipped 4 previous similar messages [10306.153554] Lustre: server umount lustre-MDT0000 complete [10310.042563] LustreError: 321414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447257 with bad export cookie 14591696193577223229 [10310.053549] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10310.058422] LustreError: 321414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10310.490500] Lustre: server umount lustre-MDT0001 complete [10324.530393] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [10327.779434] Key type lgssc unregistered [10328.033531] LNet: 327701:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10328.047141] LNetError: 327701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10328.073440] LNet: Removed LNI 192.168.201.133@tcp [10329.142505] Key type .llcrypt unregistered [10329.148931] Key type ._llcrypt unregistered [10354.040246] Key type ._llcrypt registered [10354.042269] Key type .llcrypt registered [10354.156633] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [10355.536717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10355.591059] alg: No test for adler32 (adler32-zlib) [10356.717198] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [10356.892196] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [10358.599843] Key type lgssc registered [10359.754135] Lustre: Echo OBD driver; http://www.lustre.org/ [10367.540689] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10367.595061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10369.199096] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10374.313272] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10379.761268] LustreError: 329130:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10379.784950] LustreError: 329130:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10383.004811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10383.220640] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10387.958192] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10394.744550] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10400.158789] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10403.808252] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10403.817167] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10403.826202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10404.852325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10404.855728] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10407.970213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10407.974532] Lustre: Skipped 2 previous similar messages [10413.029752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10413.044487] Lustre: Skipped 1 previous similar message [10418.143349] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10418.145659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10418.180112] Lustre: Skipped 1 previous similar message [10418.360985] Lustre: server umount lustre-MDT0000 complete [10421.566231] LustreError: 329110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447368 with bad export cookie 2917245922258752873 [10421.571116] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10421.574455] LustreError: 329110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10421.822119] Lustre: server umount lustre-MDT0001 complete [10436.753351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10437.141485] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10437.173467] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10437.325707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10442.306583] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10451.031191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10451.174827] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10451.240385] Lustre: 331716:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10451.456532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10456.288543] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10464.459151] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10470.264789] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10479.534063] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10479.801603] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10479.811211] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10479.814306] LustreError: 333600:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [10479.819862] LustreError: 333600:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [10479.827312] LustreError: 333600:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10479.891460] Lustre: server umount lustre-OST0000 complete [10479.894601] LustreError: 333600:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10490.409360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10490.648628] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10490.908131] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10496.895565] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10497.196482] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10504.791951] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10515.052036] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10515.300345] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10515.394941] Lustre: lustre-OST0001: new disk, initializing [10515.398901] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10515.410449] Lustre: 335142:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10515.524332] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10522.279920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10522.288315] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10522.400758] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10522.871861] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10530.967064] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10537.963990] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10537.964544] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10537.973426] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10538.007093] Lustre: Skipped 1 previous similar message [10548.191163] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10548.342687] Lustre: server umount lustre-OST0001 complete [10553.318962] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10553.323083] LustreError: 334067:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10553.326854] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10553.326863] LustreError: Skipped 1 previous similar message [10553.337044] Lustre: Skipped 1 previous similar message [10553.350517] LustreError: 334067:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10558.448877] Lustre: server umount lustre-OST0000 complete [10563.553257] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10563.556229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10563.560211] Lustre: Skipped 1 previous similar message [10563.566087] Lustre: Skipped 8 previous similar messages [10567.816911] Lustre: server umount lustre-MDT0000 complete [10568.675185] LustreError: 331726:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10568.714051] LustreError: 331726:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10571.896482] LustreError: 331710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447519 with bad export cookie 2917245922258753433 [10571.917687] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10572.125448] Lustre: server umount lustre-MDT0001 complete [10584.952955] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [10587.916502] Key type lgssc unregistered [10588.239313] LNet: 337099:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10588.250023] LNetError: 337099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10588.277406] LNet: Removed LNI 192.168.201.133@tcp [10589.261693] Key type .llcrypt unregistered [10589.264053] Key type ._llcrypt unregistered [10667.713958] Key type ._llcrypt registered [10667.715366] Key type .llcrypt registered [10667.965757] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [10685.471084] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [10687.770290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10687.798297] alg: No test for adler32 (adler32-zlib) [10689.055222] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [10689.323438] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [10691.007370] Key type lgssc registered [10692.359886] Lustre: Echo OBD driver; http://www.lustre.org/ [10728.633964] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10728.656468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10729.897304] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10729.922434] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10730.047462] Lustre: lustre-MDT0000: new disk, initializing [10730.163424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10730.197047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10735.154306] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10746.214266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10746.328773] Lustre: 342047:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10746.378930] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10746.385252] Lustre: Skipped 1 previous similar message [10746.459115] Lustre: lustre-MDT0001: new disk, initializing [10746.546486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10746.593637] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10746.607941] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10751.094974] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10760.312456] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10766.205962] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10772.979265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10773.323525] Lustre: lustre-OST0000: new disk, initializing [10773.329916] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10773.341657] Lustre: 344013:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10773.438264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10779.326649] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10779.703575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10779.711874] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10779.808341] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10789.786710] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10795.030854] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10795.216482] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10799.291521] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10799.473248] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10805.223917] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10805.234818] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10805.261461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10805.759812] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10805.775240] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10805.787853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10807.343972] Lustre: server umount lustre-OST0000 complete [10813.412181] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10813.418278] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10813.433822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10815.968374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10815.969314] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10817.221086] LustreError: 345555:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (000000003593d66f) refcount nonzero (2) after lock cleanup; forcing cleanup. [10819.332673] Lustre: server umount lustre-MDT0000 complete [10822.716583] LustreError: 342039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447769 with bad export cookie 14242376544837760152 [10822.720904] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10822.731783] LustreError: 342039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10823.072910] Lustre: server umount lustre-MDT0001 complete [10832.509201] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 11:02:57 (1788447777) [10840.578978] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 11:03:06 (1788447786) [10849.211835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10849.628352] LustreError: 346770:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10849.654173] LustreError: 346770:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10849.717430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10854.341112] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10854.887462] LustreError: 346771:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10858.984875] LustreError: 346770:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10863.729247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10864.236168] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10869.331961] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10878.778337] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10885.616979] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10893.375878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10893.806795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10900.901452] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10907.869248] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10916.388685] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10918.435938] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10920.043353] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10922.168523] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10924.444825] Lustre: Failing over lustre-MDT0000 [10924.523895] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10924.550617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10924.553764] Lustre: Skipped 1 previous similar message [10924.780343] Lustre: server umount lustre-MDT0000 complete [10925.908723] LustreError: 349362:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10930.999732] LustreError: 346767:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10931.022203] LustreError: 346767:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10939.874784] LustreError: 346771:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10939.885165] LustreError: 346771:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [10943.272748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10943.436842] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10943.679496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10943.735038] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10946.405227] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10948.204420] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [10949.103515] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10949.135264] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10949.192039] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10959.006908] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10960.638584] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10964.526385] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10966.543788] Lustre: Failing over lustre-MDT0000 [10966.714918] Lustre: server umount lustre-MDT0000 complete [10966.871368] LustreError: 346765:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10966.881439] LustreError: 346765:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10969.569137] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10969.570522] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10969.575647] Lustre: Skipped 3 previous similar messages [10984.929855] Lustre: 340257:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788447916/real 1788447916] req@000000008b181af5 x1875323268455296/t0(0) o400->MGC192.168.201.133@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788447932 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10984.964731] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10986.970850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10996.718072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10996.840470] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10997.058484] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11001.673915] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11001.847982] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11001.852348] Lustre: Skipped 2 previous similar messages [11001.913214] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [11001.979935] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11012.660720] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11014.028269] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11022.306544] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11022.321080] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11022.323097] Lustre: Skipped 1 previous similar message [11022.324763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11022.324770] Lustre: Skipped 2 previous similar messages [11027.230309] Lustre: server umount lustre-MDT0000 complete [11027.430193] LustreError: 346765:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11027.447152] LustreError: 346765:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 29 previous similar messages [11030.701697] LustreError: 346750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788447977 with bad export cookie 14242376544837762861 [11030.706525] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11030.714396] LustreError: 346750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11031.150339] Lustre: server umount lustre-MDT0001 complete [11045.749639] Lustre: server umount lustre-OST0000 complete [11059.601318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11065.836509] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11074.682173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11074.946730] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11074.951860] Lustre: Skipped 1 previous similar message [11079.006369] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11085.916515] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11090.883496] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11098.351876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11105.023032] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11107.769835] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11116.252361] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11120.610633] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11120.611792] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11120.615610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11120.615621] Lustre: Skipped 2 previous similar messages [11120.630163] Lustre: Skipped 3 previous similar messages [11126.656584] Lustre: server umount lustre-OST0000 complete [11137.821910] Lustre: server umount lustre-MDT0000 complete [11141.097622] LustreError: 353111:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11141.107656] LustreError: 353111:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11142.126034] LustreError: 353096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448089 with bad export cookie 14242376544837763603 [11142.129295] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11142.151580] LustreError: 353096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11142.751528] Lustre: server umount lustre-MDT0001 complete [11153.421963] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [11156.405232] Key type lgssc unregistered [11156.652257] LNet: 356644:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11156.657364] LNetError: 356644:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11156.679069] LNet: Removed LNI 192.168.201.133@tcp [11157.633700] Key type .llcrypt unregistered [11157.638595] Key type ._llcrypt unregistered [11184.168602] Key type ._llcrypt registered [11184.170754] Key type .llcrypt registered [11184.247251] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [11185.104859] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11185.252577] alg: No test for adler32 (adler32-zlib) [11186.415630] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11186.656096] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [11188.391230] Key type lgssc registered [11189.396725] Lustre: Echo OBD driver; http://www.lustre.org/ [11196.291464] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11196.311364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11197.593053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11201.154809] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11207.153081] LustreError: 358159:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11207.169844] LustreError: 358159:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11208.744148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11209.336882] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11214.792664] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11224.295239] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11230.451567] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11235.053277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11235.316509] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11240.975833] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11242.475851] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11249.598270] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11265.008719] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11265.012061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11265.019258] Lustre: Skipped 2 previous similar messages [11265.037349] Lustre: Skipped 2 previous similar messages [11269.226862] Lustre: server umount lustre-MDT0000 complete [11270.117249] LustreError: 358159:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11270.136979] LustreError: 358159:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11273.808784] LustreError: 358139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448221 with bad export cookie 13660446004795046930 [11273.825974] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11274.225160] Lustre: server umount lustre-MDT0001 complete [11286.956277] Lustre: server umount lustre-OST0000 complete [11300.920451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11301.465122] LustreError: 361894:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11301.545476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11306.665191] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11306.978293] LustreError: 361895:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11311.084171] LustreError: 361894:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11316.970857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11317.346264] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11322.663935] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11330.636715] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11336.728410] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11345.729752] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11345.900537] LustreError: 363715:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11345.920401] LustreError: 363715:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11346.272574] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11351.557204] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11353.578354] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11362.210723] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11378.153722] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11378.161515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11378.171074] Lustre: Skipped 1 previous similar message [11378.193546] Lustre: Skipped 2 previous similar messages [11381.700925] Lustre: server umount lustre-MDT0000 complete [11383.266746] LustreError: 361891:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11383.294650] LustreError: 361891:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11386.156438] LustreError: 361874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448333 with bad export cookie 13660446004795048043 [11386.171145] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11386.174488] LustreError: 361874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [11386.645107] Lustre: server umount lustre-MDT0001 complete [11401.544914] Lustre: server umount lustre-OST0000 complete [11421.138732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11421.503305] LustreError: 365805:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11421.584101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11429.505946] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11440.235637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11445.538382] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11454.404708] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11462.225625] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11470.860239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11471.397323] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11471.403744] Lustre: Skipped 1 previous similar message [11476.889547] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11485.581958] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11490.865775] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11495.907096] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11495.909635] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11495.919153] LustreError: Skipped 1 previous similar message [11495.921705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11495.964582] Lustre: Skipped 2 previous similar messages [11497.419481] Lustre: server umount lustre-OST0000 complete [11501.026314] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11501.045877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11501.057840] Lustre: Skipped 2 previous similar messages [11506.146224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11515.359141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11515.500747] Lustre: server umount lustre-MDT0000 complete [11516.387155] LustreError: 365806:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11516.411815] LustreError: 365806:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11519.590202] LustreError: 366540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448466 with bad export cookie 13660446004795049065 [11519.596596] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11519.604477] LustreError: 366540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11520.135305] Lustre: server umount lustre-MDT0001 complete [11532.437604] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [11535.681370] Key type lgssc unregistered [11535.939069] LNet: 369339:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11535.943749] LNetError: 369339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11535.962694] LNet: Removed LNI 192.168.201.133@tcp [11536.971625] Key type .llcrypt unregistered [11536.973586] Key type ._llcrypt unregistered [11563.717843] Key type ._llcrypt registered [11563.719810] Key type .llcrypt registered [11563.851593] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [11565.385488] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11565.426913] alg: No test for adler32 (adler32-zlib) [11566.460619] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11566.665706] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [11568.375168] Key type lgssc registered [11569.649366] Lustre: Echo OBD driver; http://www.lustre.org/ [11577.645157] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11577.673106] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11577.681928] LustreError: 370777:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11577.697340] LustreError: 370777:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11577.706264] LustreError: 370777:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11577.714794] LustreError: 370777:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11577.722900] LustreError: 370777:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11585.217482] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11585.234600] LustreError: 371174:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11585.262207] LustreError: 371174:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11585.268212] LustreError: 371174:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11585.273101] LustreError: 371174:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11585.284585] LustreError: 371174:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11603.371978] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [11606.223726] Key type lgssc unregistered [11606.553281] LNet: 372177:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11606.562241] LNetError: 372177:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11606.575662] LNet: Removed LNI 192.168.201.133@tcp [11607.560537] Key type .llcrypt unregistered [11607.562440] Key type ._llcrypt unregistered [11626.215250] Key type ._llcrypt registered [11626.216886] Key type .llcrypt registered [11626.343682] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [11641.594679] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [11642.718908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11642.783123] alg: No test for adler32 (adler32-zlib) [11643.912909] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [11644.117274] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [11645.769253] Key type lgssc registered [11647.085936] Lustre: Echo OBD driver; http://www.lustre.org/ [11681.744682] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11681.793174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11683.187843] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11683.230174] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11683.343372] Lustre: lustre-MDT0000: new disk, initializing [11683.479854] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11683.515069] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11688.650048] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11700.198385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11700.385599] Lustre: 375331:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11700.452249] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11700.457262] Lustre: Skipped 1 previous similar message [11700.554124] Lustre: lustre-MDT0001: new disk, initializing [11700.663801] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11700.684097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11700.697480] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11705.651746] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11715.853811] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11721.338631] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11728.991747] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11729.292453] Lustre: lustre-OST0000: new disk, initializing [11729.298469] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11729.307223] Lustre: 377300:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11729.450375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11737.190681] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11739.171835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11739.182609] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11739.292162] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11747.667991] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11752.777190] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11753.082721] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11757.659628] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11757.805058] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11760.101533] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11760.111857] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11760.134474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11763.173670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11763.176977] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11765.975732] Lustre: server umount lustre-OST0000 complete [11772.395993] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11772.404186] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11772.435138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11772.445768] Lustre: Skipped 1 previous similar message [11776.341673] Lustre: server umount lustre-MDT0000 complete [11780.105460] LustreError: 375322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448727 with bad export cookie 1844793639706416524 [11780.110553] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11780.118270] LustreError: 375322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11780.483811] Lustre: server umount lustre-MDT0001 complete [11789.586729] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11791.503788] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 11:18:56 (1788448736) [11798.671442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11799.052297] LustreError: 379758:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11799.061528] LustreError: 379758:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11799.135787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11803.141801] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11804.132242] LustreError: 379759:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11809.262542] LustreError: 379758:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11811.690383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11811.991550] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11816.362344] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11824.679472] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11831.050585] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11838.381099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11838.768734] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11845.822521] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11854.249995] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11861.916309] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11862.080143] Lustre: lustre-OST0001: new disk, initializing [11862.082973] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11862.112056] Lustre: 382595:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11862.206897] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11869.036403] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [11869.904071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11869.911744] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11870.071860] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11879.291332] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11885.538792] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11885.543638] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11885.558568] Lustre: Skipped 1 previous similar message [11885.565305] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11885.565324] Lustre: Skipped 2 previous similar messages [11889.287266] Lustre: server umount lustre-OST0001 complete [11890.671069] LustreError: 381579:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11890.692699] LustreError: 381579:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11895.781750] LustreError: 381579:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11895.799628] LustreError: 381579:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11906.018241] LustreError: 381582:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11906.043159] LustreError: 381582:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11908.580198] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11908.586108] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11908.606704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11908.613374] Lustre: Skipped 2 previous similar messages [11912.475671] Lustre: server umount lustre-OST0000 complete [11919.329257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11919.342939] LustreError: Skipped 1 previous similar message [11919.348034] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11919.367132] Lustre: Skipped 1 previous similar message [11919.381105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11919.386405] Lustre: Skipped 3 previous similar messages [11931.104703] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11931.366778] Lustre: server umount lustre-MDT0000 complete [11931.618832] LustreError: 379759:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11931.644327] LustreError: 379759:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11936.167705] LustreError: 379741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788448883 with bad export cookie 1844793639706417406 [11936.169545] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11936.175202] LustreError: 379741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11942.780438] Lustre: server umount lustre-MDT0001 complete [11954.125459] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [11957.520899] Key type lgssc unregistered [11958.002295] LNet: 384582:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11958.013402] LNetError: 384582:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11958.045844] LNet: Removed LNI 192.168.201.133@tcp [11959.001111] Key type .llcrypt unregistered [11959.003189] Key type ._llcrypt unregistered [12012.854766] Key type ._llcrypt registered [12012.856281] Key type .llcrypt registered [12012.950795] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [12014.254967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12014.430408] alg: No test for adler32 (adler32-zlib) [12015.508933] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12015.716434] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [12017.417054] Key type lgssc registered [12018.691227] Lustre: Echo OBD driver; http://www.lustre.org/ [12026.522708] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12026.535921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12027.946193] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12027.971946] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12028.174710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12032.617586] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [12042.599887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12042.736820] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12042.793223] Lustre: 387377:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12043.066306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12047.385355] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [12055.021251] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12060.521035] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12063.712470] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12063.719251] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12063.742168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12066.272491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12066.300954] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12069.011240] Lustre: server umount lustre-MDT0000 complete [12072.903332] LustreError: 388120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449020 with bad export cookie 12921968331564235735 [12072.904653] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12072.911421] LustreError: 388120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12073.248279] Lustre: server umount lustre-MDT0001 complete [12083.705201] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 11:23:49 (1788449029) [12090.881993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12091.359196] LustreError: 389893:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12091.378849] LustreError: 389893:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12091.426946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12096.497620] LustreError: 389894:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12097.425523] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [12101.600117] LustreError: 389893:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12106.738608] LustreError: 389894:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12108.091761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12108.764628] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12114.062879] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [12121.110693] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12126.731214] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12132.991165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12133.131856] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12133.295349] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12138.246864] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [12147.738842] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12156.142509] Lustre: Permanently deactivating lustre-OST0000 [12156.156570] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12165.146484] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12166.909591] LustreError: 392926:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12166.916315] LustreError: 392926:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12166.921540] LustreError: 392926:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12169.238615] LustreError: 392975:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12169.261379] LustreError: 392975:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12177.492222] Lustre: server umount lustre-OST0000 complete [12180.218805] LustreError: 393224:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12180.226133] LustreError: 393224:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12185.570146] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12185.577175] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12185.586159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12185.590703] Lustre: Skipped 1 previous similar message [12191.201291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12191.207818] Lustre: Skipped 3 previous similar messages [12192.280056] Lustre: server umount lustre-MDT0000 complete [12196.088570] LustreError: 389873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449143 with bad export cookie 12921968331564236309 [12196.102223] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12196.105444] LustreError: 389873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12206.712534] Lustre: server umount lustre-MDT0001 complete [12217.468540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12223.094679] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [12226.355820] LustreError: 394385:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12232.416246] Lustre: 394723:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.133@tcp [12234.375951] Lustre: server umount MGS complete [12245.561439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12246.066219] LustreError: 395448:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12246.077838] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12246.094975] Lustre: Skipped 1 previous similar message [12246.153468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12250.635223] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [12256.741211] LustreError: 395448:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12256.764819] LustreError: 395448:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12259.273725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12259.651659] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12264.564862] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [12271.200501] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12277.021369] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12279.683691] Lustre: Permanently reactivating lustre-OST0000 [12279.720301] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12279.723210] Lustre: Skipped 2 previous similar messages [12308.959219] LustreError: 397230:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12315.924116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12316.327954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12316.335363] Lustre: Skipped 1 previous similar message [12323.353865] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [12333.439477] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12339.174805] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12339.186367] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12339.194405] Lustre: Skipped 1 previous similar message [12339.210753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12341.176520] Lustre: server umount lustre-OST0000 complete [12346.851504] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12346.857911] LustreError: Skipped 1 previous similar message [12346.862335] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12346.883514] Lustre: Skipped 1 previous similar message [12350.432475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12350.435547] Lustre: Skipped 3 previous similar messages [12351.834579] Lustre: server umount lustre-MDT0000 complete [12355.556245] LustreError: 395445:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12355.583333] LustreError: 395445:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12355.796550] LustreError: 396186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449303 with bad export cookie 12921968331564237366 [12355.804690] LustreError: 396186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12355.812656] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12356.110942] Lustre: server umount lustre-MDT0001 complete [12367.149787] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [12369.861876] Key type lgssc unregistered [12370.146166] LNet: 399812:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12370.152931] LNetError: 399812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12370.165657] LNet: Removed LNI 192.168.201.133@tcp [12370.890508] Key type .llcrypt unregistered [12370.893301] Key type ._llcrypt unregistered [12388.354798] Key type ._llcrypt registered [12388.357199] Key type .llcrypt registered [12388.426531] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [12405.096374] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [12406.023107] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12406.097824] alg: No test for adler32 (adler32-zlib) [12407.523862] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12408.011109] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [12409.863178] Key type lgssc registered [12411.397650] Lustre: Echo OBD driver; http://www.lustre.org/ [12451.026407] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12451.083679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12452.503456] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12452.546478] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12452.739607] Lustre: lustre-MDT0000: new disk, initializing [12452.901348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12452.924578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12458.096739] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [12470.836391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12470.964039] Lustre: 402963:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12471.020846] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12471.024825] Lustre: Skipped 1 previous similar message [12471.111539] Lustre: lustre-MDT0001: new disk, initializing [12471.268040] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12471.337928] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12471.349698] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12476.108963] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [12486.146589] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12493.236393] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12500.495387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12500.767971] Lustre: lustre-OST0000: new disk, initializing [12500.775822] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12500.783991] Lustre: 404931:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12500.893970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12506.442916] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [12506.687581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12506.701133] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12506.794292] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12515.698877] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12519.857856] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12520.049870] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12524.180455] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12524.353418] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12527.585766] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12527.593194] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12527.596461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12527.611790] Lustre: Skipped 1 previous similar message [12532.709203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12532.718708] Lustre: Skipped 2 previous similar messages [12537.829710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12537.838979] Lustre: Skipped 1 previous similar message [12540.383128] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12540.648682] Lustre: server umount lustre-OST0000 complete [12548.065741] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12548.082765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12548.084448] Lustre: Skipped 1 previous similar message [12548.102640] Lustre: Skipped 1 previous similar message [12553.191819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12553.197988] Lustre: Skipped 1 previous similar message [12558.305071] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12558.448500] Lustre: server umount lustre-MDT0000 complete [12561.991911] LustreError: 402954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449509 with bad export cookie 18182111749299055791 [12561.998254] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12562.358652] Lustre: server umount lustre-MDT0001 complete [12570.462983] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12572.412672] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 11:31:57 (1788449517) [12580.508663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12581.324491] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12586.465791] LustreError: 407396:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12586.477323] LustreError: 407396:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12586.544594] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [12591.592923] LustreError: 407397:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12596.727549] LustreError: 407396:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12597.536840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12597.858547] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12603.058683] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [12612.548489] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12619.537529] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12626.686158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12627.007231] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12633.835810] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [12644.554804] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12647.091960] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12654.272719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12661.559647] Lustre: server umount lustre-OST0000 complete [12666.851020] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12666.857167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12666.867060] Lustre: Skipped 1 previous similar message [12666.876245] Lustre: Skipped 3 previous similar messages [12675.099492] Lustre: server umount lustre-MDT0000 complete [12677.089358] LustreError: 407393:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12677.108178] LustreError: 407393:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12678.113952] Lustre: 401186:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788449608/real 1788449608] req@000000002a343964 x1875325070940160/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788449624 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12678.147223] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12678.351467] LustreError: 407377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449625 with bad export cookie 18182111749299056673 [12678.376164] LustreError: 407377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12678.385885] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12678.630468] Lustre: server umount lustre-MDT0001 complete [12688.279462] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [12690.460524] Key type lgssc unregistered [12690.705087] LNet: 411118:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12690.713991] LNetError: 411118:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12691.749917] LNet: Removed LNI 192.168.201.133@tcp [12692.573540] Key type .llcrypt unregistered [12692.577092] Key type ._llcrypt unregistered [12715.147323] Key type ._llcrypt registered [12715.152104] Key type .llcrypt registered [12715.231164] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [12715.625874] Key type .llcrypt unregistered [12715.627942] Key type ._llcrypt unregistered [12735.257344] Key type ._llcrypt registered [12735.258862] Key type .llcrypt registered [12735.347741] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [12736.295250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12736.405346] alg: No test for adler32 (adler32-zlib) [12737.536972] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12737.711134] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [12739.399249] Key type lgssc registered [12740.811494] Lustre: Echo OBD driver; http://www.lustre.org/ [12749.918247] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12749.943224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12751.307824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12755.807919] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [12761.581301] LustreError: 413320:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12761.604793] LustreError: 413320:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12766.702897] LustreError: 413321:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12767.795921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12768.350064] LustreError: 413320:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12768.405928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12773.866803] LustreError: 413316:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12773.893714] LustreError: 413316:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12775.923199] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12777.989963] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [12781.024734] LustreError: 413315:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12781.056632] LustreError: 413315:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12786.785230] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12791.284788] LustreError: 413316:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12791.313262] LustreError: 413316:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12796.278738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12796.826529] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12801.128302] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [12811.744849] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12811.749329] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12811.758423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12812.277055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12812.288427] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12812.295109] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12817.380331] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12817.390835] Lustre: Skipped 1 previous similar message [12817.898270] Lustre: server umount lustre-OST0000 complete [12822.497813] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12822.500948] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12822.501963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12822.501971] Lustre: Skipped 1 previous similar message [12822.505619] Lustre: Skipped 1 previous similar message [12824.013055] Lustre: server umount lustre-MDT0000 complete [12827.215953] LustreError: 413302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449774 with bad export cookie 11166830961162955035 [12827.220764] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12827.222441] LustreError: 413302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12827.543592] Lustre: server umount lustre-MDT0001 complete [12838.639104] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [12841.580989] Key type lgssc unregistered [12841.940256] LNet: 416468:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12841.948455] LNetError: 416468:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12841.962904] LNet: Removed LNI 192.168.201.133@tcp [12843.088462] Key type .llcrypt unregistered [12843.090996] Key type ._llcrypt unregistered [12876.496480] Key type ._llcrypt registered [12876.500618] Key type .llcrypt registered [12876.668323] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [12878.330946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12878.365178] alg: No test for adler32 (adler32-zlib) [12879.680322] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [12880.106095] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [12881.871182] Key type lgssc registered [12883.507257] Lustre: Echo OBD driver; http://www.lustre.org/ [12892.880421] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12892.952642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12908.767764] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12913.887439] LustreError: 417861:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.133@tcp: failed processing log, type 4: rc = -110 [12940.640131] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12947.736233] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [12957.918094] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12965.065543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12965.580674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12965.640212] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12970.308877] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [12970.987496] LustreError: 417887:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12971.021078] LustreError: 417887:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12975.085155] LustreError: 417886:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12975.113529] LustreError: 417886:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12980.199402] LustreError: 418899:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12980.220880] LustreError: 418899:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12980.385234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12980.716208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12985.625693] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [12992.849841] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12999.557293] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13007.877674] Lustre: server umount lustre-OST0000 complete [13015.016768] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [13015.020178] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13015.027493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13016.544812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13016.554599] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13021.671879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13021.678789] Lustre: Skipped 2 previous similar messages [13022.764429] Lustre: server umount lustre-MDT0000 complete [13026.784408] Lustre: 417195:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788449958/real 1788449958] req@000000009bc11fb2 x1875325565827840/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788449974 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13026.800868] LustreError: 418903:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13026.816284] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13026.847615] LustreError: 418903:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13027.049617] LustreError: 417868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788449974 with bad export cookie 16318707380778015929 [13027.059835] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13027.424647] Lustre: server umount lustre-MDT0001 complete [13039.706172] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [13042.728832] Key type lgssc unregistered [13043.134657] LNet: 421429:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13043.145744] LNetError: 421429:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13043.173731] LNet: Removed LNI 192.168.201.133@tcp [13044.223439] Key type .llcrypt unregistered [13044.224941] Key type ._llcrypt unregistered [13070.732943] Key type ._llcrypt registered [13070.734840] Key type .llcrypt registered [13070.830692] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [13072.002321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13072.115206] alg: No test for adler32 (adler32-zlib) [13073.258389] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [13073.485459] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [13075.151225] Key type lgssc registered [13076.532822] Lustre: Echo OBD driver; http://www.lustre.org/ [13086.421735] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13086.464114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13088.250527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13094.535890] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [13098.488725] LustreError: 422842:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13098.501541] LustreError: 422842:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13103.592978] LustreError: 422843:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13105.303492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13105.714814] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13110.716565] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [13119.352383] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13126.529203] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13134.300968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13134.870352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13142.733411] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [13151.598462] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13158.465191] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13167.073048] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13167.080501] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13167.093989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13167.962763] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.33@tcp (stopping) [13168.615844] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13168.630951] Lustre: Skipped 1 previous similar message [13172.049856] Lustre: server umount lustre-MDT0000 complete [13173.060979] LustreError: 422839:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13174.903683] LustreError: 422822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.33@tcp arrived at 1788450122 with bad export cookie 10244567196050455918 [13177.083393] LustreError: 422823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450124 with bad export cookie 10244567196050455253 [13177.091055] LustreError: 422823:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13177.098655] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13178.850929] LustreError: 422843:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13178.857942] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13178.867048] LustreError: 422843:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13178.902577] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13178.914487] Lustre: Skipped 3 previous similar messages [13182.880513] LustreError: 425335:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13182.881455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13182.894767] LustreError: 425335:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13183.349467] Lustre: server umount lustre-OST0000 complete [13191.337947] Lustre: server umount lustre-MDT0001 complete [13204.226885] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [13207.354297] Key type lgssc unregistered [13207.708771] LNet: 426551:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13207.712197] LNetError: 426551:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13207.742768] LNet: Removed LNI 192.168.201.133@tcp [13208.845786] Key type .llcrypt unregistered [13208.850813] Key type ._llcrypt unregistered [13233.476280] Key type ._llcrypt registered [13233.482105] Key type .llcrypt registered [13233.635519] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [13234.833213] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13234.993410] alg: No test for adler32 (adler32-zlib) [13236.081888] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [13236.293085] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [13237.951685] Key type lgssc registered [13239.286599] Lustre: Echo OBD driver; http://www.lustre.org/ [13247.510574] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13247.536540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13249.061050] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13253.308041] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [13259.244250] LustreError: 427974:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13259.271064] LustreError: 427974:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13262.284284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13262.640729] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13267.501175] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [13276.011567] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13282.934784] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13290.552831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13290.999951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13294.068494] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:193) [13297.334296] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [13304.589695] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13312.312920] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.33@tcp (stopping) [13312.491678] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13315.895495] Lustre: server umount lustre-MDT0001 complete [13317.434070] LustreError: 427970:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.33@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13322.730210] LustreError: 427973:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13322.751445] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13322.785961] LustreError: 427973:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13322.797227] Lustre: Skipped 1 previous similar message [13322.812080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13322.830916] Lustre: Skipped 2 previous similar messages [13327.842604] LustreError: 427974:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13327.853469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13328.400438] Lustre: server umount lustre-OST0000 complete [13333.350406] Lustre: server umount lustre-MDT0000 complete [13349.338847] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [13352.316513] Key type lgssc unregistered [13352.751756] LNet: 431659:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13352.762362] LNetError: 431659:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13352.794302] LNet: Removed LNI 192.168.201.133@tcp [13353.803122] Key type .llcrypt unregistered [13353.805886] Key type ._llcrypt unregistered [13380.548838] Key type ._llcrypt registered [13380.551195] Key type .llcrypt registered [13380.648597] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [13396.603667] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [13398.626793] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13398.640947] alg: No test for adler32 (adler32-zlib) [13399.556059] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [13399.783996] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [13401.567151] Key type lgssc registered [13402.924471] Lustre: Echo OBD driver; http://www.lustre.org/ [13441.318620] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [13458.035718] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13458.051838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13459.324373] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13459.343938] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13459.398154] Lustre: lustre-MDT0000: new disk, initializing [13459.452046] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13459.467371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13464.787561] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [13482.253131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13482.476298] Lustre: lustre-OST0000: new disk, initializing [13482.480307] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13482.488147] Lustre: Skipped 1 previous similar message [13482.501605] Lustre: 436691:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13482.586321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13488.636263] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [13490.280621] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13490.287088] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13490.312851] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13498.189331] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13513.027639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13513.103316] Lustre: 435671:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13513.146565] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13513.204378] Lustre: lustre-MDT0001: new disk, initializing [13513.298329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13513.334285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13513.343224] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13517.921100] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [13526.346402] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13526.658715] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13531.092941] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13531.253648] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13540.841083] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13540.852379] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13540.862918] Lustre: Skipped 1 previous similar message [13540.881196] Lustre: Skipped 1 previous similar message [13543.208286] Lustre: server umount lustre-MDT0001 complete [13551.086693] LustreError: 435683:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13551.104959] LustreError: 435683:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [13556.199081] LustreError: 437403:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13556.220129] LustreError: 437403:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13561.313323] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13561.317770] LustreError: 435684:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13561.328566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13561.348142] Lustre: Skipped 1 previous similar message [13566.439551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13566.440498] LustreError: 435683:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13567.534631] Lustre: server umount lustre-OST0000 complete [13585.375552] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13585.591595] Lustre: server umount lustre-MDT0000 complete [13603.156976] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [13606.235862] Key type lgssc unregistered [13606.463759] LNet: 440440:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13606.471530] LNetError: 440440:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13606.483311] LNet: Removed LNI 192.168.201.133@tcp [13607.335753] Key type .llcrypt unregistered [13607.337886] Key type ._llcrypt unregistered [13648.400778] Key type ._llcrypt registered [13648.405881] Key type .llcrypt registered [13648.555777] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [13648.993897] Key type .llcrypt unregistered [13649.010082] Key type ._llcrypt unregistered [13665.995755] Key type ._llcrypt registered [13665.997769] Key type .llcrypt registered [13666.100865] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [13667.077609] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13667.103922] alg: No test for adler32 (adler32-zlib) [13668.306865] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [13668.594277] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [13670.303196] Key type lgssc registered [13671.336868] Lustre: Echo OBD driver; http://www.lustre.org/ [13698.805605] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13698.859439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13700.154489] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13700.217582] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13700.374823] Lustre: lustre-MDT0000: new disk, initializing [13700.529591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13700.559243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13705.708287] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [13716.247505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13716.356278] Lustre: 443808:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13716.397286] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13716.400459] Lustre: Skipped 1 previous similar message [13716.455387] Lustre: lustre-MDT0001: new disk, initializing [13716.616829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13716.698419] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13716.710200] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13722.443257] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [13732.772616] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13739.289697] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13747.537466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13747.781662] Lustre: lustre-OST0000: new disk, initializing [13747.785984] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13747.791136] Lustre: 445777:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13747.878123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13753.431038] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13753.439440] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13753.618606] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13755.173985] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [13766.229954] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13774.311738] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13774.326335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13774.330281] Lustre: Skipped 1 previous similar message [13774.346579] Lustre: Skipped 2 previous similar messages [13779.427198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13779.433212] Lustre: Skipped 2 previous similar messages [13780.289971] Lustre: server umount lustre-MDT0000 complete [13783.682616] LustreError: 443802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450730 with bad export cookie 14408218287024053946 [13783.705206] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13783.705723] LustreError: 443802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13784.111235] Lustre: server umount lustre-MDT0001 complete [13798.989890] Lustre: server umount lustre-OST0000 complete [13809.372736] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 11:52:35 (1788450755) [13817.193239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13817.688085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13822.336524] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [13822.947071] LustreError: 447808:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13822.959674] LustreError: 447808:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13827.040439] LustreError: 447809:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13830.762503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13831.212614] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13834.756260] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [13842.041358] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13848.520387] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13870.457406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13870.851889] Lustre: Found index 0 for lustre-OST0000, updating log [13871.119976] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13878.594415] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [13887.169764] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13888.578406] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13896.672743] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13896.687270] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13896.697779] Lustre: Skipped 1 previous similar message [13896.707693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13903.043598] Lustre: server umount lustre-OST0000 complete [13919.210731] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13919.219476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13919.233100] Lustre: Skipped 2 previous similar messages [13919.252613] Lustre: Skipped 4 previous similar messages [13924.323242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13924.336842] Lustre: Skipped 1 previous similar message [13925.253265] Lustre: server umount lustre-MDT0000 complete [13925.347372] LustreError: 447809:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13925.363970] LustreError: 447809:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13929.292683] LustreError: 447788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788450876 with bad export cookie 14408218287024055339 [13929.295351] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13929.310143] LustreError: 447788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [13929.646933] Lustre: server umount lustre-MDT0001 complete [13931.811096] LNet: 1 local NIs in recovery (showing 1): 192.168.201.133@tcp [13940.820959] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 11:54:45 (1788450885) [14562.532714] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14563.086072] LustreError: 454236:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14563.100907] LustreError: 454236:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14563.186244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14568.207786] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [14568.427488] LustreError: 454237:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14573.545338] LustreError: 454236:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14577.447714] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14578.052410] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14582.900411] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [14591.444040] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14598.376555] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14620.401527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14620.704218] Lustre: Found index 0 for lustre-OST0000, updating log [14621.008490] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14623.012551] LNet: 1 local NIs in recovery (showing 1): 192.168.201.133@tcp [14628.383452] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [14637.452370] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14640.688022] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14640.704584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14645.987978] Lustre: server umount lustre-OST0000 complete [14651.042627] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14651.052472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14651.060504] Lustre: Skipped 1 previous similar message [14653.089201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14653.095020] Lustre: Skipped 3 previous similar messages [14656.377843] Lustre: server umount lustre-MDT0000 complete [14657.191029] LustreError: 454237:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14657.202244] LustreError: 454237:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14660.390407] LustreError: 456086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788451607 with bad export cookie 14408218287024056424 [14660.399054] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14660.399376] LustreError: 456086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14660.752505] Lustre: server umount lustre-MDT0001 complete [14683.423313] LNet: 1 local NIs in recovery (showing 1): 192.168.201.133@tcp [14692.097843] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [14704.207548] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [14746.647864] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 12:08:11 (1788451691) [14767.841929] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [14771.354429] Key type lgssc unregistered [14771.813777] LNet: 460850:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14771.821443] LNetError: 460850:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14882.408811] LNet: Removed LNI 192.168.201.133@tcp [14883.685495] Key type .llcrypt unregistered [14883.691704] Key type ._llcrypt unregistered [14902.233760] Key type ._llcrypt registered [14902.242869] Key type .llcrypt registered [14902.400123] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [14903.823058] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14904.076244] alg: No test for adler32 (adler32-zlib) [14905.257537] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [14905.481413] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [14907.135461] Key type lgssc registered [14908.490952] Lustre: Echo OBD driver; http://www.lustre.org/ [14913.932403] LNet: Added LNI 192.168.201.133@tcp100 [8/256/0/180] [14914.954759] LNet: Added LNI 192.168.201.133@tcp101 [8/256/0/180] [14915.890094] LNet: Added LNI 192.168.201.133@tcp102 [8/256/0/180] [14916.788058] LNet: Added LNI 192.168.201.133@tcp103 [8/256/0/180] [14917.799619] LNet: Added LNI 192.168.201.133@tcp104 [8/256/0/180] [14918.809885] LNet: Added LNI 192.168.201.133@tcp105 [8/256/0/180] [14919.814703] LNet: Added LNI 192.168.201.133@tcp106 [8/256/0/180] [14920.829598] LNet: Added LNI 192.168.201.133@tcp107 [8/256/0/180] [14921.765220] LNet: Added LNI 192.168.201.133@tcp108 [8/256/0/180] [14922.616699] LNet: Added LNI 192.168.201.133@tcp109 [8/256/0/180] [14923.630259] LNet: Added LNI 192.168.201.133@tcp110 [8/256/0/180] [14924.764748] LNet: Added LNI 192.168.201.133@tcp111 [8/256/0/180] [14925.695388] LNet: Added LNI 192.168.201.133@tcp112 [8/256/0/180] [14926.595595] LNet: Added LNI 192.168.201.133@tcp113 [8/256/0/180] [14927.655697] LNet: Added LNI 192.168.201.133@tcp114 [8/256/0/180] [14928.863971] LNet: Added LNI 192.168.201.133@tcp115 [8/256/0/180] [14929.883474] LNet: Added LNI 192.168.201.133@tcp116 [8/256/0/180] [14930.865882] LNet: Added LNI 192.168.201.133@tcp117 [8/256/0/180] [14931.817446] LNet: Added LNI 192.168.201.133@tcp118 [8/256/0/180] [14932.950833] LNet: Added LNI 192.168.201.133@tcp119 [8/256/0/180] [14934.046039] LNet: Added LNI 192.168.201.133@tcp120 [8/256/0/180] [14935.107421] LNet: Added LNI 192.168.201.133@tcp121 [8/256/0/180] [14936.127315] LNet: Added LNI 192.168.201.133@tcp122 [8/256/0/180] [14937.081067] LNet: Added LNI 192.168.201.133@tcp123 [8/256/0/180] [14938.108887] LNet: Added LNI 192.168.201.133@tcp124 [8/256/0/180] [14939.165162] LNet: Added LNI 192.168.201.133@tcp125 [8/256/0/180] [14940.146890] LNet: Added LNI 192.168.201.133@tcp126 [8/256/0/180] [14940.998979] LNet: Added LNI 192.168.201.133@tcp127 [8/256/0/180] [14942.097816] LNet: Added LNI 192.168.201.133@tcp128 [8/256/0/180] [14943.204413] LNet: Added LNI 192.168.201.133@tcp129 [8/256/0/180] [14944.290874] LNet: Added LNI 192.168.201.133@tcp130 [8/256/0/180] [14945.397694] LNet: Added LNI 192.168.201.133@tcp131 [8/256/0/180] [14946.576253] LNet: Added LNI 192.168.201.133@tcp132 [8/256/0/180] [14947.732035] LNet: Added LNI 192.168.201.133@tcp133 [8/256/0/180] [14948.747844] LNet: Added LNI 192.168.201.133@tcp134 [8/256/0/180] [14949.857154] LNet: Added LNI 192.168.201.133@tcp135 [8/256/0/180] [14957.890544] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14957.910644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14959.084515] Lustre: 463879:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.133@tcp: can't add NID 192.168.201.133@tcp130: rc = -75 [14959.224288] LNetError: 463902:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.133 [14959.238492] LNetError: Protocol error connecting to 192.168.201.133@tcp119 at host 192.168.201.133:988: is it running a compatible version of Lustre? [14959.252115] LNetError: Skipped 4 previous similar messages [14959.342852] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14959.394661] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14959.513899] Lustre: lustre-MDT0000: new disk, initializing [14959.626941] Lustre: 463934:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.133@tcp: can't add NID 192.168.201.133@tcp130: rc = -75 [14959.639144] Lustre: 463934:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14959.661558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14959.682143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14959.782588] LNet: 7 local NIs in recovery (showing 7): 192.168.201.133@tcp124, 192.168.201.133@tcp126, 192.168.201.133@tcp125, 192.168.201.133@tcp128, 192.168.201.133@tcp129, 192.168.201.133@tcp119, 192.168.201.133@tcp127 [14960.290675] LNetError: 463899:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.133 [14960.318595] LNetError: 463899:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 10 previous similar messages [14960.329653] LNetError: Protocol error connecting to 192.168.201.133@tcp115 at host 192.168.201.133:988: is it running a compatible version of Lustre? [14960.341085] LNetError: Skipped 3 previous similar messages [14962.915206] LNetError: 463899:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.133 [14962.932621] LNetError: 463899:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [14962.949592] LNetError: Protocol error connecting to 192.168.201.133@tcp124 at host 192.168.201.133:988: is it running a compatible version of Lustre? [14962.966522] LNetError: Skipped 8 previous similar messages [14965.665520] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [14967.009411] LNetError: 463903:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.133 [14967.017546] LNetError: 463903:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [14967.028469] LNetError: Protocol error connecting to 192.168.201.133@tcp124 at host 192.168.201.133:988: is it running a compatible version of Lustre? [14975.205443] LNetError: 463904:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.133 [14975.219807] LNetError: Protocol error connecting to 192.168.201.133@tcp124 at host 192.168.201.133:988: is it running a compatible version of Lustre? [14977.031598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14977.244537] Lustre: 463895:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14977.326057] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14977.331254] Lustre: Skipped 1 previous similar message [14977.484515] Lustre: lustre-MDT0001: new disk, initializing [14977.530860] Lustre: 464696:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.133@tcp: can't add NID 192.168.201.133@tcp130: rc = -75 [14977.540383] Lustre: 464696:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14977.730373] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14977.767392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14977.786495] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14982.707364] Lustre: 464936:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.133@tcp: can't add NID 192.168.201.133@tcp130: rc = -75 [14982.713600] Lustre: 464936:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14983.038814] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [14991.533474] LNetError: 463911:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.133 [14991.547412] LNetError: Protocol error connecting to 192.168.201.133@tcp122 at host 192.168.201.133:988: is it running a compatible version of Lustre? [14994.712535] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15002.241336] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15007.907411] LNetError: 461338:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.133 [15007.919881] LNetError: 461338:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 8 previous similar messages [15007.927246] LNetError: Protocol error connecting to 192.168.201.133@tcp120 at host 192.168.201.133:988: is it running a compatible version of Lustre? [15007.942648] LNetError: Skipped 8 previous similar messages [15009.092400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15009.465250] Lustre: lustre-OST0000: new disk, initializing [15009.469103] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15009.477946] Lustre: 465869:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15009.586940] Lustre: 465874:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.133@tcp: can't add NID 192.168.201.133@tcp130: rc = -75 [15009.595214] Lustre: 465874:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15009.662945] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15015.516242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15015.546720] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15015.707036] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15016.480730] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [15020.255417] LNet: 1 local NIs in recovery (showing 1): 192.168.201.133@tcp124 [15026.511511] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15036.391896] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15036.402777] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15036.408981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15036.417686] Lustre: Skipped 1 previous similar message [15036.434541] Lustre: Skipped 1 previous similar message [15039.837691] Lustre: server umount lustre-OST0000 complete [15040.674260] LNetError: 461337:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.133 [15040.688732] LNetError: 461337:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 14 previous similar messages [15040.699581] LNetError: Protocol error connecting to 192.168.201.133@tcp119 at host 192.168.201.133:988: is it running a compatible version of Lustre? [15040.709688] LNetError: Skipped 14 previous similar messages [15044.066464] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15044.069091] LustreError: Skipped 1 previous similar message [15044.071657] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15044.080224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15046.629320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15046.641992] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15051.748088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15051.759989] Lustre: Skipped 2 previous similar messages [15055.842986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15055.858990] Lustre: Skipped 1 previous similar message [15058.399478] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15058.661205] Lustre: server umount lustre-MDT0000 complete [15062.488834] LustreError: 463889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452009 with bad export cookie 4475321676799225300 [15062.497705] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15062.510788] LustreError: 463889:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15062.746189] Lustre: server umount lustre-MDT0001 complete [15075.596181] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [15079.188664] Key type lgssc unregistered [15079.892649] LNet: 467785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15079.931965] LNetError: 467785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15079.948476] LNet: Removed LNI 192.168.201.133@tcp [15080.991226] LNet: Removed LNI 192.168.201.133@tcp100 [15080.997195] LNetError: 467785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15081.002858] LNetError: 467785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15082.079183] LNet: Removed LNI 192.168.201.133@tcp101 [15082.081929] LNetError: 467785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15082.094029] LNet: Removed LNI 192.168.201.133@tcp102 [15083.103832] LNet: Removed LNI 192.168.201.133@tcp103 [15084.127602] LNet: Removed LNI 192.168.201.133@tcp104 [15084.129884] LNetError: 467785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15084.134874] LNetError: 467785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15084.142120] LNet: Removed LNI 192.168.201.133@tcp105 [15085.155153] LNet: Removed LNI 192.168.201.133@tcp106 [15086.179958] LNet: Removed LNI 192.168.201.133@tcp107 [15087.199270] LNet: Removed LNI 192.168.201.133@tcp108 [15088.225340] LNet: Removed LNI 192.168.201.133@tcp109 [15088.233884] LNetError: 467785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15088.250762] LNetError: 467785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15088.263340] LNet: Removed LNI 192.168.201.133@tcp110 [15089.311215] LNet: Removed LNI 192.168.201.133@tcp111 [15090.335225] LNet: Removed LNI 192.168.201.133@tcp112 [15091.359554] LNet: Removed LNI 192.168.201.133@tcp113 [15092.384447] LNet: Removed LNI 192.168.201.133@tcp114 [15093.407157] LNet: Removed LNI 192.168.201.133@tcp115 [15094.431202] LNet: Removed LNI 192.168.201.133@tcp116 [15095.457370] LNet: Removed LNI 192.168.201.133@tcp117 [15096.479347] LNet: Removed LNI 192.168.201.133@tcp118 [15096.492495] LNetError: 467785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15096.509464] LNetError: 467785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15096.523477] LNet: Removed LNI 192.168.201.133@tcp119 [15097.567687] LNet: Removed LNI 192.168.201.133@tcp120 [15098.659070] LNet: Removed LNI 192.168.201.133@tcp121 [15099.679175] LNet: Removed LNI 192.168.201.133@tcp122 [15100.703551] LNet: Removed LNI 192.168.201.133@tcp123 [15100.712067] LNet: Removed LNI 192.168.201.133@tcp124 [15101.727198] LNet: Removed LNI 192.168.201.133@tcp125 [15102.751557] LNet: Removed LNI 192.168.201.133@tcp126 [15103.775495] LNet: Removed LNI 192.168.201.133@tcp127 [15104.799405] LNet: Removed LNI 192.168.201.133@tcp128 [15105.823624] LNet: Removed LNI 192.168.201.133@tcp129 [15105.827204] LNet: Removed LNI 192.168.201.133@tcp130 [15105.834566] LNet: Removed LNI 192.168.201.133@tcp131 [15105.841691] LNet: Removed LNI 192.168.201.133@tcp132 [15105.844890] LNet: Removed LNI 192.168.201.133@tcp133 [15105.851860] LNet: Removed LNI 192.168.201.133@tcp134 [15105.871916] LNet: Removed LNI 192.168.201.133@tcp135 [15106.753735] Key type .llcrypt unregistered [15106.756302] Key type ._llcrypt unregistered [15130.951724] Key type ._llcrypt registered [15130.955588] Key type .llcrypt registered [15131.096982] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [15148.372959] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [15149.927335] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15150.067811] alg: No test for adler32 (adler32-zlib) [15151.162987] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [15151.399658] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [15153.127220] Key type lgssc registered [15154.611171] Lustre: Echo OBD driver; http://www.lustre.org/ [15193.731843] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 12:15:39 (1788452139) [15214.125861] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [15217.372717] Key type lgssc unregistered [15217.701927] LNet: 471872:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15217.715727] LNetError: 471872:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15217.738941] LNet: Removed LNI 192.168.201.133@tcp [15218.684867] Key type .llcrypt unregistered [15218.687438] Key type ._llcrypt unregistered [15235.803592] Key type ._llcrypt registered [15235.805233] Key type .llcrypt registered [15235.890440] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [15236.723250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15236.774765] alg: No test for adler32 (adler32-zlib) [15237.966719] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [15238.207759] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [15239.919140] Key type lgssc registered [15240.996099] Lustre: Echo OBD driver; http://www.lustre.org/ [15249.712989] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15249.738642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15250.936116] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15250.959379] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15251.037209] Lustre: lustre-MDT0000: new disk, initializing [15251.110968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15251.128729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15256.062524] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [15269.443530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15269.623267] Lustre: 473082:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15269.678453] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15269.681315] Lustre: Skipped 1 previous similar message [15269.739179] Lustre: lustre-MDT0001: new disk, initializing [15269.831984] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15269.881765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15269.902136] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15276.414918] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [15287.821488] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15294.777356] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15302.555960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15302.796269] Lustre: lustre-OST0000: new disk, initializing [15302.801437] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15302.806590] Lustre: 475052:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15302.889317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15308.459420] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [15311.930260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15311.954582] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15312.036468] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15318.220611] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15323.420449] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing lnet_if_list [15330.432740] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15330.459205] LNet: Added LNI 192.168.201.133@tcp10 [8/256/0/180] [15337.067715] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15337.130491] LNet: Added LNI 192.168.201.133@tcp11 [8/256/0/180] [15341.429296] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15341.452680] LNet: Added LNI 192.168.201.133@tcp12 [8/256/0/180] [15346.687450] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15346.711022] LNet: Added LNI 192.168.201.133@tcp13 [8/256/0/180] [15352.029835] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15352.060763] LNet: Added LNI 192.168.201.133@tcp14 [8/256/0/180] [15357.879317] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15357.902255] LNet: Added LNI 192.168.201.133@tcp15 [8/256/0/180] [15362.860770] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15362.904850] LNet: Added LNI 192.168.201.133@tcp16 [8/256/0/180] [15368.249890] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15368.274713] LNet: Added LNI 192.168.201.133@tcp17 [8/256/0/180] [15373.568102] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15373.596760] LNet: Added LNI 192.168.201.133@tcp18 [8/256/0/180] [15378.274389] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15378.313898] LNet: Added LNI 192.168.201.133@tcp19 [8/256/0/180] [15384.600794] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15384.628469] LNet: Added LNI 192.168.201.133@tcp20 [8/256/0/180] [15389.888569] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15389.927688] LNet: Added LNI 192.168.201.133@tcp21 [8/256/0/180] [15395.339397] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15395.372349] LNet: Added LNI 192.168.201.133@tcp22 [8/256/0/180] [15402.361150] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15402.410070] LNet: Added LNI 192.168.201.133@tcp23 [8/256/0/180] [15410.541501] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15410.561673] LNet: Added LNI 192.168.201.133@tcp24 [8/256/0/180] [15418.582212] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15418.615944] LNet: Added LNI 192.168.201.133@tcp25 [8/256/0/180] [15424.915922] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15424.941864] LNet: Added LNI 192.168.201.133@tcp26 [8/256/0/180] [15430.549732] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15430.576649] LNet: Added LNI 192.168.201.133@tcp27 [8/256/0/180] [15436.952486] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15436.982079] LNet: Added LNI 192.168.201.133@tcp28 [8/256/0/180] [15443.079629] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15443.099859] LNet: Added LNI 192.168.201.133@tcp29 [8/256/0/180] [15450.763534] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15450.782422] LNet: Added LNI 192.168.201.133@tcp30 [8/256/0/180] [15458.396466] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15458.421221] LNet: Added LNI 192.168.201.133@tcp31 [8/256/0/180] [15464.434236] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15464.453831] LNet: Added LNI 192.168.201.133@tcp32 [8/256/0/180] [15470.298937] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15470.336244] LNet: Added LNI 192.168.201.133@tcp33 [8/256/0/180] [15477.135414] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15477.158033] LNet: Added LNI 192.168.201.133@tcp34 [8/256/0/180] [15484.066575] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15484.094697] LNet: Added LNI 192.168.201.133@tcp35 [8/256/0/180] [15491.099485] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15491.123260] LNet: Added LNI 192.168.201.133@tcp36 [8/256/0/180] [15499.569834] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15499.601848] LNet: Added LNI 192.168.201.133@tcp37 [8/256/0/180] [15509.331426] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15509.401028] LNet: Added LNI 192.168.201.133@tcp38 [8/256/0/180] [15521.430715] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15521.488159] LNet: Added LNI 192.168.201.133@tcp39 [8/256/0/180] [15530.726403] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15530.773269] LNet: Added LNI 192.168.201.133@tcp40 [8/256/0/180] [15539.787291] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15539.817438] LNet: Added LNI 192.168.201.133@tcp41 [8/256/0/180] [15547.607778] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15547.649133] LNet: Added LNI 192.168.201.133@tcp42 [8/256/0/180] [15556.482399] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15556.502483] LNet: Added LNI 192.168.201.133@tcp43 [8/256/0/180] [15564.842763] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15564.885538] LNet: Added LNI 192.168.201.133@tcp44 [8/256/0/180] [15573.000714] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15573.024291] LNet: Added LNI 192.168.201.133@tcp45 [8/256/0/180] [15581.766217] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15581.808807] LNet: Added LNI 192.168.201.133@tcp46 [8/256/0/180] [15590.989958] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15591.022476] LNet: Added LNI 192.168.201.133@tcp47 [8/256/0/180] [15599.620807] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15599.644329] LNet: Added LNI 192.168.201.133@tcp48 [8/256/0/180] [15609.243730] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15609.267714] LNet: Added LNI 192.168.201.133@tcp49 [8/256/0/180] [15619.104682] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15619.134798] LNet: Added LNI 192.168.201.133@tcp50 [8/256/0/180] [15643.108546] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15643.109879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15643.127613] Lustre: Skipped 2 previous similar messages [15643.143198] Lustre: Skipped 2 previous similar messages [15645.610904] Lustre: server umount lustre-MDT0000 complete [15649.184330] LustreError: 480884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452596 with bad export cookie 18036478254809173409 [15649.187108] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15649.195059] LustreError: 480884:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15649.458443] Lustre: server umount lustre-MDT0001 complete [15668.344485] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [15669.676034] Lustre: 472353:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788452600/real 1788452600] req@00000000a5de6649 x1875328038523520/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788452616 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15669.727789] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15670.754310] Lustre: 472355:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788452601/real 1788452601] req@00000000c7b6882c x1875328038523648/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788452617 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15673.896866] Lustre: 472354:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788452605/real 1788452605] req@0000000011c72822 x1875328038523776/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788452621 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15678.320781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15679.967321] Lustre: 472352:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788452611/real 1788452611] req@0000000069bf4b63 x1875328038524160/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788452627 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15680.000036] Lustre: 472352:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [15694.306197] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xfa4e716950f557f0 [15694.319162] Lustre: MGC192.168.201.133@tcp: Connection restored to 0@lo (at 0@lo) [15694.784354] LustreError: 475045:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15694.813194] LustreError: 475045:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [15694.885726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15697.952610] LustreError: 475046:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15697.962266] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15700.005473] LustreError: 475045:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15701.076725] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [15705.070152] LustreError: 475046:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15710.177794] LustreError: 475045:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15710.718700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15711.310421] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15714.435814] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15717.063990] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [15727.300200] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15733.971233] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15755.748657] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15755.773539] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15760.871987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15760.886107] Lustre: Skipped 2 previous similar messages [15765.984780] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15769.055860] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15769.299581] Lustre: server umount lustre-OST0000 complete [15776.225860] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15776.229113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15776.237818] Lustre: Skipped 2 previous similar messages [15776.253667] Lustre: Skipped 2 previous similar messages [15779.913739] Lustre: server umount lustre-MDT0000 complete [15781.346411] LustreError: 499297:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15781.362811] LustreError: 499297:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15783.743408] LustreError: 473072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452730 with bad export cookie 18036478254809176048 [15783.747236] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15783.751302] LustreError: 473072:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15784.221853] Lustre: server umount lustre-MDT0001 complete [15795.497538] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [15798.358548] Key type lgssc unregistered [15798.824039] LNet: 502102:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15798.833433] LNetError: 502102:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15799.839155] LNet: Removed LNI 192.168.201.133@tcp [15799.844108] LNetError: 502102:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15799.850735] LNet: Removed LNI 192.168.201.133@tcp10 [15799.855121] LNet: Removed LNI 192.168.201.133@tcp11 [15799.858547] LNet: Removed LNI 192.168.201.133@tcp12 [15799.862145] LNet: Removed LNI 192.168.201.133@tcp13 [15799.866546] LNet: Removed LNI 192.168.201.133@tcp14 [15799.879082] LNet: Removed LNI 192.168.201.133@tcp15 [15799.890785] LNet: Removed LNI 192.168.201.133@tcp16 [15799.897169] LNet: Removed LNI 192.168.201.133@tcp17 [15799.911827] LNet: Removed LNI 192.168.201.133@tcp18 [15799.928102] LNet: Removed LNI 192.168.201.133@tcp19 [15799.937256] LNet: Removed LNI 192.168.201.133@tcp20 [15799.949751] LNet: Removed LNI 192.168.201.133@tcp21 [15799.956977] LNet: Removed LNI 192.168.201.133@tcp22 [15799.962328] LNet: Removed LNI 192.168.201.133@tcp23 [15799.964570] LNet: Removed LNI 192.168.201.133@tcp24 [15799.971637] LNet: Removed LNI 192.168.201.133@tcp25 [15799.982288] LNet: Removed LNI 192.168.201.133@tcp26 [15799.983959] LNet: Removed LNI 192.168.201.133@tcp27 [15799.988101] LNet: Removed LNI 192.168.201.133@tcp28 [15799.992641] LNet: Removed LNI 192.168.201.133@tcp29 [15799.995344] LNet: Removed LNI 192.168.201.133@tcp30 [15800.005394] LNet: Removed LNI 192.168.201.133@tcp31 [15800.013179] LNet: Removed LNI 192.168.201.133@tcp32 [15800.016331] LNet: Removed LNI 192.168.201.133@tcp33 [15800.020252] LNet: Removed LNI 192.168.201.133@tcp34 [15800.025254] LNet: Removed LNI 192.168.201.133@tcp35 [15800.028444] LNet: Removed LNI 192.168.201.133@tcp36 [15800.037177] LNet: Removed LNI 192.168.201.133@tcp37 [15800.046228] LNet: Removed LNI 192.168.201.133@tcp38 [15800.064088] LNet: Removed LNI 192.168.201.133@tcp39 [15800.074855] LNet: Removed LNI 192.168.201.133@tcp40 [15800.085203] LNet: Removed LNI 192.168.201.133@tcp41 [15800.093322] LNet: Removed LNI 192.168.201.133@tcp42 [15800.106280] LNet: Removed LNI 192.168.201.133@tcp43 [15800.113832] LNet: Removed LNI 192.168.201.133@tcp44 [15800.124930] LNet: Removed LNI 192.168.201.133@tcp45 [15800.132875] LNet: Removed LNI 192.168.201.133@tcp46 [15800.135159] LNet: Removed LNI 192.168.201.133@tcp47 [15800.137639] LNet: Removed LNI 192.168.201.133@tcp48 [15800.139886] LNet: Removed LNI 192.168.201.133@tcp49 [15800.169990] LNet: Removed LNI 192.168.201.133@tcp50 [15800.939365] Key type .llcrypt unregistered [15800.941635] Key type ._llcrypt unregistered [15823.403679] Key type ._llcrypt registered [15823.408231] Key type .llcrypt registered [15823.620647] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [15839.845625] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [15841.639449] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15841.667617] alg: No test for adler32 (adler32-zlib) [15843.099688] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [15843.573910] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [15845.343217] Key type lgssc registered [15847.043264] Lustre: Echo OBD driver; http://www.lustre.org/ [15887.617552] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15887.666152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15889.045919] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15889.087591] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15889.285108] Lustre: lustre-MDT0000: new disk, initializing [15889.458212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15889.491544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15894.893851] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [15910.042283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15910.198849] Lustre: 505297:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15910.410155] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15910.426851] Lustre: Skipped 1 previous similar message [15910.548622] Lustre: lustre-MDT0001: new disk, initializing [15910.670219] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15910.712352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15910.727084] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15916.646289] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [15928.878663] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15935.643898] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15943.617408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15944.009784] Lustre: *** cfs_fail_loc=90f, val=0*** [15944.160744] Lustre: lustre-OST0000: new disk, initializing [15944.165149] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15944.172745] Lustre: 507316:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15944.432285] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15952.546122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15952.574459] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15952.796235] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15953.411969] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [15967.674245] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15978.466843] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15978.473704] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15978.496354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15978.506646] Lustre: Skipped 1 previous similar message [15978.519184] Lustre: Skipped 1 previous similar message [15983.600331] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15983.613186] Lustre: Skipped 1 previous similar message [15988.725936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15988.733068] Lustre: Skipped 1 previous similar message [15991.263210] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15991.724393] Lustre: server umount lustre-OST0000 complete [15997.932347] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15997.950754] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15997.969855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16003.202343] Lustre: server umount lustre-MDT0000 complete [16007.845209] LustreError: 506094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788452955 with bad export cookie 171052524027115971 [16007.846030] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16007.853551] LustreError: 506094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16008.289572] Lustre: server umount lustre-MDT0001 complete [16021.503469] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [16024.686730] Key type lgssc unregistered [16025.001745] LNet: 509223:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16025.018183] LNetError: 509223:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16025.038168] LNet: Removed LNI 192.168.201.133@tcp [16026.247079] Key type .llcrypt unregistered [16026.249062] Key type ._llcrypt unregistered [16052.056973] Key type ._llcrypt registered [16052.066640] Key type .llcrypt registered [16052.194609] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [16053.545807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16053.617210] alg: No test for adler32 (adler32-zlib) [16055.403406] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [16055.818239] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [16057.559231] Key type lgssc registered [16059.459098] Lustre: Echo OBD driver; http://www.lustre.org/ [16071.003927] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16071.017401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16072.681073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16078.690762] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [16081.900841] LustreError: 510493:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16081.932286] LustreError: 510493:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [16087.009158] LustreError: 510494:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16089.469650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16090.051852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16096.087049] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [16104.800445] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16111.780897] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16121.076227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16122.092552] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16131.349468] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [16140.545314] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16158.469272] loop0: detected capacity change from 0 to 204800000 [16158.646133] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16158.682914] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16158.705822] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16165.929494] loop0: detected capacity change from 0 to 204800000 [16166.218345] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16166.262543] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16166.286521] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16172.528728] loop0: detected capacity change from 0 to 204800000 [16180.078685] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16180.345271] Lustre: fs15246-OST0000: new disk, initializing [16180.356722] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16180.364640] Lustre: 514100:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16180.443344] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16189.036941] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [16204.747941] loop1: detected capacity change from 0 to 204800000 [16210.717460] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16211.194095] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16211.438376] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16211.630704] Lustre: fs15246-MDT0000: new disk, initializing [16212.109135] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16212.182806] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16212.521413] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16213.613942] Lustre: 515223:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000a4fd7fe3 x1875328896132096/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788453176 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16221.187438] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [16222.790770] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16222.799158] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16222.857161] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16251.364406] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16251.388311] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16254.536511] Lustre: server umount fs15246-MDT0000 complete [16256.496876] LustreError: 515647:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16261.620496] LustreError: 515647:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16265.885957] Lustre: server umount fs15246-OST0000 complete [16281.059794] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16281.072871] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16281.088850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16282.088688] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16282.113825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16282.133855] Lustre: Skipped 1 previous similar message [16286.519515] Lustre: server umount lustre-OST0000 complete [16292.334751] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16292.340213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16292.360272] Lustre: Skipped 1 previous similar message [16292.381027] Lustre: Skipped 1 previous similar message [16297.444387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16297.464976] Lustre: Skipped 1 previous similar message [16298.159391] Lustre: server umount lustre-MDT0000 complete [16302.561159] LustreError: 510489:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16302.595606] LustreError: 510489:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [16302.849460] LustreError: 516367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453250 with bad export cookie 8440262488038420402 [16302.872307] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16302.873713] LustreError: 516367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [16303.481568] Lustre: server umount lustre-MDT0001 complete [16317.138697] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [16320.443760] Key type lgssc unregistered [16320.775398] LNet: 518326:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16320.803800] LNetError: 518326:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16320.824876] LNet: Removed LNI 192.168.201.133@tcp [16322.252792] Key type .llcrypt unregistered [16322.257330] Key type ._llcrypt unregistered [16343.992770] Key type ._llcrypt registered [16343.995369] Key type .llcrypt registered [16344.104588] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [16345.433503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16345.702045] alg: No test for adler32 (adler32-zlib) [16346.833878] Lustre: Lustre: Build Version: 2.17.58_2_gee80f9b [16347.135113] LNet: Added LNI 192.168.201.133@tcp [8/256/0/180] [16348.855669] Key type lgssc registered [16350.232184] Lustre: Echo OBD driver; http://www.lustre.org/ [16400.038536] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [16410.368604] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [16454.226331] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16454.242954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16455.597948] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16455.681646] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16455.776287] Lustre: lustre-MDT0000: new disk, initializing [16456.058163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16456.106664] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16462.299340] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [16475.185379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16475.323686] Lustre: 522795:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16475.377470] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16475.382382] Lustre: Skipped 1 previous similar message [16475.505300] Lustre: lustre-MDT0001: new disk, initializing [16475.606092] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16475.644891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16475.659337] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16481.012391] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [16492.784077] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16500.429991] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16509.244799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16509.596312] Lustre: lustre-OST0000: new disk, initializing [16509.600853] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16509.605789] Lustre: 524763:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16509.736444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16517.714479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16517.726654] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16517.731897] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_default_debug -1 all [16517.871156] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16529.556916] Lustre: DEBUG MARKER: oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16534.958986] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16535.261600] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16541.505186] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16541.711941] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16548.837536] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16548.853080] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16548.864417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16553.954675] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16553.963038] Lustre: Skipped 1 previous similar message [16558.047361] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16558.295720] Lustre: server umount lustre-OST0000 complete [16564.192843] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16564.200245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16564.222836] Lustre: Skipped 2 previous similar messages [16564.255301] Lustre: Skipped 2 previous similar messages [16569.313654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16569.338829] Lustre: Skipped 1 previous similar message [16571.664138] Lustre: server umount lustre-MDT0000 complete [16575.910797] LustreError: 522785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788453523 with bad export cookie 5571575506246444551 [16575.914396] LustreError: MGC192.168.201.133@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16575.922914] LustreError: 522785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16576.699420] Lustre: server umount lustre-MDT0001 complete [16609.214239] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing set_hostid [16623.900321] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing load_modules_local [16663.216860] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16368 sec ========= 12:40:08 (1788453608) [16665.249616] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:40:10 (1788453610) === [16669.199537] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:40:14 (1788453614) === [16716.313031] Lustre: DEBUG MARKER: oleg133-server.virtnet: executing unload_modules_local [16720.202483] Key type lgssc unregistered [16720.469925] LNet: 531413:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16720.476451] LNetError: 531413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16721.515640] LNet: Removed LNI 192.168.201.133@tcp [16722.619926] Key type .llcrypt unregistered [16722.623060] Key type ._llcrypt unregistered