[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 477100325 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002307] x2apic enabled [ 0.003011] Switched APIC routing to physical x2apic. [ 0.004010] kvm-guest: setup PV IPIs [ 0.007328] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008029] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009010] pid_max: default: 32768 minimum: 301 [ 0.010125] LSM: Security Framework initializing [ 0.011043] Yama: becoming mindful. [ 0.012032] SELinux: Initializing. [ 0.013054] *** VALIDATE selinux *** [ 0.021505] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026487] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028038] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029103] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030134] *** VALIDATE tmpfs *** [ 0.032469] *** VALIDATE proc *** [ 0.033247] *** VALIDATE cgroup *** [ 0.034009] *** VALIDATE cgroup2 *** [ 0.035240] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036169] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037018] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038034] Spectre V2 : User space: Vulnerable [ 0.039007] Speculative Store Bypass: Vulnerable [ 0.042392] debug: unmapping init [mem 0xffffffff90059000-0xffffffff90060fff] [ 0.044180] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045679] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046019] ... version: 2 [ 0.046890] ... bit width: 48 [ 0.047011] ... generic registers: 4 [ 0.048009] ... value mask: 0000ffffffffffff [ 0.049009] ... max period: 00007fffffffffff [ 0.050013] ... fixed-purpose events: 3 [ 0.051018] ... event mask: 000000070000000f [ 0.052468] rcu: Hierarchical SRCU implementation. [ 0.054625] smp: Bringing up secondary CPUs ... [ 0.055768] x86: Booting SMP configuration: [ 0.056045] .... node #0, CPUs: #1 #2 #3 [ 0.065252] smp: Brought up 1 node, 4 CPUs [ 0.067022] smpboot: Max logical packages: 1 [ 0.068025] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.145126] node 0 deferred pages initialised in 75ms [ 0.148011] devtmpfs: initialized [ 0.149252] x86/mm: Memory block size: 128MB [ 0.151441] gcov: version magic: 0x41383552 [ 0.154359] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.155090] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.156324] pinctrl core: initialized pinctrl subsystem [ 0.157155] [ 0.157735] ************************************************************* [ 0.158012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.159013] ** ** [ 0.160009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.161009] ** ** [ 0.162013] ** This means that this kernel is built to expose internal ** [ 0.163015] ** IOMMU data structures, which may compromise security on ** [ 0.164014] ** your system. ** [ 0.165010] ** ** [ 0.166013] ** If you see this message and you are not debugging the ** [ 0.167016] ** kernel, report this immediately to your vendor! ** [ 0.168014] ** ** [ 0.169011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.170009] ************************************************************* [ 0.171645] NET: Registered protocol family 16 [ 0.173484] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.176072] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.178060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.182113] cpuidle: using governor menu [ 0.183570] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.185320] PCI: Using configuration type 1 for base access [ 0.186121] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.193137] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.195067] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.197056] cryptd: max_cpu_qlen set to 1000 [ 0.199212] ACPI: Added _OSI(Module Device) [ 0.200008] ACPI: Added _OSI(Processor Device) [ 0.200970] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.201007] ACPI: Added _OSI(Processor Aggregator Device) [ 0.204893] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.209432] ACPI: Interpreter enabled [ 0.210041] ACPI: PM: (supports S0 S3 S4 S5) [ 0.210906] ACPI: Using IOAPIC for interrupt routing [ 0.212105] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.214493] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.222180] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.224023] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.225009] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.227063] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.229938] acpiphp: Slot [2] registered [ 0.231080] acpiphp: Slot [5] registered [ 0.231915] acpiphp: Slot [6] registered [ 0.232073] acpiphp: Slot [7] registered [ 0.233075] acpiphp: Slot [8] registered [ 0.233934] acpiphp: Slot [9] registered [ 0.235080] acpiphp: Slot [10] registered [ 0.236066] acpiphp: Slot [3] registered [ 0.236940] acpiphp: Slot [4] registered [ 0.238054] acpiphp: Slot [11] registered [ 0.238869] acpiphp: Slot [12] registered [ 0.239071] acpiphp: Slot [13] registered [ 0.239890] acpiphp: Slot [14] registered [ 0.241071] acpiphp: Slot [15] registered [ 0.241921] acpiphp: Slot [16] registered [ 0.243050] acpiphp: Slot [17] registered [ 0.243911] acpiphp: Slot [18] registered [ 0.245050] acpiphp: Slot [19] registered [ 0.245867] acpiphp: Slot [20] registered [ 0.247132] acpiphp: Slot [21] registered [ 0.248070] acpiphp: Slot [22] registered [ 0.249063] acpiphp: Slot [23] registered [ 0.250000] acpiphp: Slot [24] registered [ 0.250000] acpiphp: Slot [25] registered [ 0.250000] acpiphp: Slot [26] registered [ 0.252086] acpiphp: Slot [27] registered [ 0.253043] acpiphp: Slot [28] registered [ 0.254046] acpiphp: Slot [29] registered [ 0.255000] acpiphp: Slot [30] registered [ 0.255000] acpiphp: Slot [31] registered [ 0.256016] PCI host bridge to bus 0000:00 [ 0.256982] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.259017] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.260015] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.262016] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.264022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.265024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.267148] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.268849] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.271010] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.279773] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.284796] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.287016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.288011] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.290011] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.291483] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.293607] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.296037] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.298356] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.302000] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.316014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.320010] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.324451] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.329025] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.336018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.353029] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.370596] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.378021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.383022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.401014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.411434] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.421025] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.436027] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.471026] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.481027] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.493025] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.501018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.524021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.535827] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.554018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.573021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.593266] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.603456] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.611021] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.622032] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.655027] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.670747] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.674535] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.677514] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.680534] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.683404] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.690192] iommu: Default domain type: Passthrough [ 0.692632] SCSI subsystem initialized [ 0.694185] ACPI: bus type USB registered [ 0.696127] usbcore: registered new interface driver usbfs [ 0.698255] usbcore: registered new interface driver hub [ 0.701095] usbcore: registered new device driver usb [ 0.703194] pps_core: LinuxPPS API ver. 1 registered [ 0.705012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.709066] PTP clock support registered [ 0.711192] EDAC MC: Ver: 3.0.0 [ 0.712614] PCI: Using ACPI for IRQ routing [ 0.715042] NetLabel: Initializing [ 0.716015] NetLabel: domain hash size = 128 [ 0.718010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.719082] NetLabel: unlabeled traffic allowed by default [ 0.722130] vgaarb: loaded [ 0.723290] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.726017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.734667] clocksource: Switched to clocksource kvm-clock [ 0.855205] VFS: Disk quotas dquot_6.6.0 [ 0.857106] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.859977] *** VALIDATE ramfs *** [ 0.861604] *** VALIDATE hugetlbfs *** [ 0.863616] pnp: PnP ACPI init [ 0.866652] pnp: PnP ACPI: found 6 devices [ 0.908539] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.912423] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.914871] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.917226] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.920263] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.923018] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.926442] NET: Registered protocol family 2 [ 0.929124] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.934427] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.938706] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.945468] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.949230] TCP: Hash tables configured (established 65536 bind 65536) [ 0.952812] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.956665] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.960235] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.963817] NET: Registered protocol family 1 [ 0.967693] RPC: Registered named UNIX socket transport module. [ 0.970367] RPC: Registered udp transport module. [ 0.972255] RPC: Registered tcp transport module. [ 0.974115] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.976813] NET: Registered protocol family 44 [ 0.979316] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.981779] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.984385] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.987526] PCI: CLS 0 bytes, default 64 [ 0.989524] Unpacking initramfs... [ 2.416492] debug: unmapping init [mem 0xffff9f023cc54000-0xffff9f023ffbffff] [ 2.420391] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.422580] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.425535] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.947775] Initialise system trusted keyrings [ 2.949430] Key type blacklist registered [ 2.950647] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.960745] zbud: loaded [ 2.963840] *** VALIDATE nfs *** [ 2.964603] *** VALIDATE nfs4 *** [ 2.965788] pstore: using deflate compression [ 2.968476] Platform Keyring initialized [ 3.066291] NET: Registered protocol family 38 [ 3.067755] Key type asymmetric registered [ 3.068761] Asymmetric key parser 'x509' registered [ 3.069779] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.072370] io scheduler mq-deadline registered [ 3.073796] io scheduler kyber registered [ 3.075153] io scheduler bfq registered [ 3.077544] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.079572] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.082314] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.085188] ACPI: Power Button [PWRF] [ 3.091140] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.099123] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.115427] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.129327] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.150310] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.179951] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.210655] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.215699] Non-volatile memory driver v1.3 [ 3.217688] Linux agpgart interface v0.103 [ 3.254416] virtio_blk virtio1: [vda] 146000 512-byte logical blocks (74.8 MB/71.3 MiB) [ 3.256835] vda: detected capacity change from 0 to 74752000 [ 3.270507] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.272892] vdb: detected capacity change from 0 to 1073741824 [ 3.286816] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.289168] vdc: detected capacity change from 0 to 2621440000 [ 3.305506] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.308561] vdd: detected capacity change from 0 to 2621440000 [ 3.323438] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.326108] vde: detected capacity change from 0 to 4294967296 [ 3.342604] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.345192] vdf: detected capacity change from 0 to 4294967296 [ 3.353074] libphy: Fixed MDIO Bus: probed [ 3.364461] usbcore: registered new interface driver usbserial_generic [ 3.366578] usbserial: USB Serial support registered for generic [ 3.368759] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.373188] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.374986] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.377078] mousedev: PS/2 mouse device common for all mice [ 3.379043] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.379604] rtc_cmos 00:05: RTC can wake from S4 [ 3.384104] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.388222] rtc_cmos 00:05: registered as rtc0 [ 3.390036] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.390522] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.395016] intel_pstate: CPU model not supported [ 3.397026] hid: raw HID events driver (C) Jiri Kosina [ 3.398361] usbcore: registered new interface driver usbhid [ 3.399420] usbhid: USB HID core driver [ 3.400401] drop_monitor: Initializing network drop monitor service [ 3.401715] Initializing XFRM netlink socket [ 3.403433] NET: Registered protocol family 10 [ 3.406234] Segment Routing with IPv6 [ 3.407222] NET: Registered protocol family 17 [ 3.409472] mpls_gso: MPLS GSO support [ 3.414925] RAS: Correctable Errors collector initialized. [ 3.416946] AVX version of gcm_enc/dec engaged. [ 3.418029] AES CTR mode by8 optimization enabled [ 3.486377] sched_clock: Marking stable (3486305665, 0)->(4300915909, -814610244) [ 3.491464] registered taskstats version 1 [ 3.493516] Loading compiled-in X.509 certificates [ 3.494837] zswap: loaded using pool lzo/zbud [ 3.516450] Key type big_key registered [ 3.527801] Key type encrypted registered [ 3.529499] ima: No TPM chip found, activating TPM-bypass! [ 3.531308] ima: Allocated hash algorithm: sha1 [ 3.532488] ima: No architecture policies found [ 3.534238] evm: Initialising EVM extended attributes: [ 3.536075] evm: security.selinux [ 3.537279] evm: security.ima [ 3.538370] evm: security.capability [ 3.539359] evm: HMAC attrs: 0x1 [ 3.541143] rtc_cmos 00:05: setting system clock to 2026-08-19 04:49:54 UTC (1787114994) [ 3.545820] debug: unmapping init [mem 0xffffffff91003000-0xffffffff911fffff] [ 3.548054] debug: unmapping init [mem 0xffffffff8fd82000-0xffffffff90058fff] [ 3.555069] Write protecting the kernel read-only data: 28672k [ 3.557281] debug: unmapping init [mem 0xffffffff8e403000-0xffffffff8e5fffff] [ 3.558889] debug: unmapping init [mem 0xffffffff8ed14000-0xffffffff8edfffff] [ 3.584787] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.590152] systemd[1]: Detected virtualization kvm. [ 3.591619] systemd[1]: Detected architecture x86-64. [ 3.593322] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.623134] systemd[1]: No hostname configured. [ 3.625186] systemd[1]: Set hostname to . [ 3.627547] random: systemd: uninitialized urandom read (16 bytes read) [ 3.630058] systemd[1]: Initializing machine ID from random generator. [ 3.738476] random: systemd: uninitialized urandom read (16 bytes read) [ 3.740102] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.743289] random: systemd: uninitialized urandom read (16 bytes read) [ 3.745872] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.749737] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Swap. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.255934] device-mapper: uevent: version 1.0.3 [ 4.257807] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.076766] random: fast init done [ 5.077231] virtio_net virtio0 ens2: renamed from eth0 [ 5.183538] scsi host0: ata_piix [ 5.198145] scsi host1: ata_piix [ 5.200374] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.203087] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.881351] dracut-initqueue[590]: RTNETLINK answers: File exists [ 10.047781] random: crng init done [ 10.049538] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.509878] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.847677] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.134299] SELinux: Disabled at runtime. [ 12.199088] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.210078] systemd[1]: Detected virtualization kvm. [ 12.212545] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.757563] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.763419] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.770182] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.775367] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.779503] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.789804] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.796384] systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Initrd File Systems. Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. Starting Apply Kernel Variables... [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target Paths. Starting Remount Root and Kernel File Systems... [ 12.969817] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting udev Coldplug all Devices... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.287408] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.743258] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.804693] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.939423] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.960253] EDAC sbridge: Ver: 1.1.2 [ 15.909607] Key type dns_resolver registered [ 16.248324] NFS: Registering the id_resolver key type [ 16.249944] Key type id_resolver registered [ 16.251376] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg151-server login: [ 42.377422] libcfs: loading out-of-tree module taints kernel. [ 42.394337] Key type ._llcrypt registered [ 42.395430] Key type .llcrypt registered [ 42.451365] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [ 50.942579] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 51.748684] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 51.755821] alg: No test for adler32 (adler32-zlib) [ 52.809808] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 53.191150] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 54.848349] Key type lgssc registered [ 57.474737] Lustre: Echo OBD driver; http://www.lustre.org/ [ 71.283126] hrtimer: interrupt took 2916139 ns [ 78.360990] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 127.388514] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 140.445516] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 140.502088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 141.917034] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 141.979251] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 142.088861] Lustre: lustre-MDT0000: new disk, initializing [ 142.194073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 142.226554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 147.316763] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 164.059737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 164.198087] Lustre: 6494:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 164.251935] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 164.256320] Lustre: Skipped 1 previous similar message [ 164.354436] Lustre: lustre-MDT0001: new disk, initializing [ 164.473653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 164.503643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 164.512611] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 169.361913] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 174.793847] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 185.718323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 185.987882] Lustre: lustre-OST0000: new disk, initializing [ 185.991914] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 185.998967] Lustre: 8433:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 186.072802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 191.569015] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 191.595040] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 191.757759] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 194.351235] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 211.932804] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 212.084064] Lustre: lustre-OST0001: new disk, initializing [ 212.089642] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 212.101947] Lustre: 9505:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 212.206522] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 219.581449] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 222.262358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 222.274396] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 222.366964] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 233.406193] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 239.919037] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 247.231265] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing check_logdir /tmp/testlogs/ [ 253.222849] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing yml_node [ 257.843731] Lustre: DEBUG MARKER: Client: 2.17.56.3 [ 260.943774] Lustre: DEBUG MARKER: MDS: 2.17.56.3 [ 263.316777] Lustre: DEBUG MARKER: OSS: 2.17.56.3 [ 264.901232] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Aug 19 00:54:14 EDT 2026 [ 284.318933] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 285.672204] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 302.562371] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 302.575211] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 302.599762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 304.098653] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 304.099895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 304.110297] Lustre: Skipped 1 previous similar message [ 304.115650] Lustre: Skipped 1 previous similar message [ 307.674056] Lustre: server umount lustre-MDT0000 complete [ 312.280642] LustreError: 6486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787115303 with bad export cookie 807333269417686572 [ 312.285642] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 312.299370] LustreError: 6486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 314.337721] LustreError: 6501:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 314.347956] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 314.356929] LustreError: 6501:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 314.357590] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 314.357597] Lustre: Skipped 2 previous similar messages [ 314.405409] Lustre: Skipped 2 previous similar messages [ 317.378523] LustreError: 8889:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 317.378687] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 317.397431] LustreError: 8889:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 317.404890] Lustre: Skipped 2 previous similar messages [ 318.885480] Lustre: server umount lustre-MDT0001 complete [ 324.827490] Lustre: server umount lustre-OST0000 complete [ 329.951365] Lustre: server umount lustre-OST0001 complete [ 339.240909] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [ 349.023525] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 391.927117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 392.265507] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 392.330307] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 392.490274] Lustre: lustre-MDT0000: new disk, initializing [ 392.608745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 392.661281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 397.883555] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 408.701240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 408.800866] Lustre: 15059:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 408.872758] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 408.878238] Lustre: Skipped 1 previous similar message [ 409.031656] Lustre: lustre-MDT0001: new disk, initializing [ 409.153729] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 409.201399] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 409.214860] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 413.877660] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 424.512167] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 431.928487] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 441.160461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 441.528776] Lustre: lustre-OST0000: new disk, initializing [ 441.535951] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 441.548885] Lustre: 17029:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 441.680425] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 443.345980] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 443.368549] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 443.618278] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 448.220764] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 458.608390] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 462.821485] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 463.062560] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 468.546392] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 468.775786] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 474.095702] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 474.101165] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 474.130412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 474.132023] Lustre: Skipped 1 previous similar message [ 474.146894] Lustre: Skipped 1 previous similar message [ 484.335489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 484.339898] Lustre: Skipped 3 previous similar messages [ 485.344164] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 485.709995] Lustre: server umount lustre-OST0000 complete [ 489.450288] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 495.844785] Lustre: server umount lustre-MDT0000 complete [ 499.690419] LustreError: 15072:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 499.706493] LustreError: 15072:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 500.653094] LustreError: 15856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787115491 with bad export cookie 807333269417688105 [ 500.672308] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 500.675111] LustreError: 15856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 501.308305] Lustre: server umount lustre-MDT0001 complete [ 506.523413] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 00:58:16 (1787115496) [ 509.487531] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 511.875397] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 00:58:21 (1787115501) [ 519.725052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 520.386428] LustreError: 19440:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 520.522030] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 525.136390] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 525.808744] LustreError: 19441:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 534.748760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 540.110140] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 549.704874] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 556.379932] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 566.610526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 574.796244] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 584.768378] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 601.057537] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 601.059061] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 601.073869] LustreError: Skipped 1 previous similar message [ 601.075817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 601.075826] Lustre: Skipped 4 previous similar messages [ 601.121063] Lustre: Skipped 2 previous similar messages [ 606.125151] Lustre: server umount lustre-OST0000 complete [ 616.803906] Lustre: server umount lustre-MDT0000 complete [ 621.545259] LustreError: 19440:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 621.580526] LustreError: 19440:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 621.693149] LustreError: 19422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787115612 with bad export cookie 807333269417688987 [ 621.700270] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 621.724363] LustreError: 19422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 622.256874] Lustre: server umount lustre-MDT0001 complete [ 636.880230] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 641.528063] Key type lgssc unregistered [ 641.826718] LNet: 22996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 641.835447] LNetError: 22996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 642.856704] LNet: Removed LNI 192.168.201.151@tcp [ 644.257212] Key type .llcrypt unregistered [ 644.262596] Key type ._llcrypt unregistered [ 670.160906] Key type ._llcrypt registered [ 670.173751] Key type .llcrypt registered [ 670.365986] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 672.071112] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 672.146972] alg: No test for adler32 (adler32-zlib) [ 673.269704] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 673.526859] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 675.264335] Key type lgssc registered [ 676.852835] Lustre: Echo OBD driver; http://www.lustre.org/ [ 687.077573] loop: module loaded [ 687.219558] loop0: detected capacity change from 0 to 204800000 [ 687.486628] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 687.504139] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 687.836952] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 694.721808] loop0: detected capacity change from 0 to 204800000 [ 695.030217] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 695.045185] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 695.059207] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 701.906588] loop0: detected capacity change from 0 to 204800000 [ 710.037501] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 710.078521] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 711.509335] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 711.629177] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 711.768688] Lustre: lustre-MDT0000: new disk, initializing [ 711.947283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 711.982275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 716.901721] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 726.766936] loop1: detected capacity change from 0 to 204800000 [ 733.361828] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 733.650558] Lustre: lustre-OST07c6: new disk, initializing [ 733.653815] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 733.657774] Lustre: Skipped 1 previous similar message [ 733.665314] Lustre: 26201:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 733.719606] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 740.911161] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 743.615406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 743.625052] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 743.666113] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 747.210125] Lustre: 25185:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 747.240438] Lustre: 25185:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 747.248345] Lustre: 25185:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 747.253867] Lustre: 25185:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 747.258366] Lustre: 25185:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 747.264168] Lustre: 25185:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 761.430176] Lustre: 25185:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 761.439791] Lustre: 25185:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 761.445659] Lustre: 25185:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 761.453304] Lustre: 25185:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 761.458465] Lustre: 25185:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 761.473864] Lustre: 25185:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 767.066909] Lustre: 25187:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 767.074092] Lustre: 25187:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 767.087614] Lustre: 25187:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 767.096127] Lustre: 25187:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 767.107542] Lustre: 25187:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 767.117698] Lustre: 25187:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 767.124585] Lustre: 25187:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 767.137509] Lustre: 25187:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 767.147820] Lustre: 25187:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 767.163940] Lustre: 25187:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 767.171505] Lustre: 25187:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 767.182556] Lustre: 25187:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 770.488921] Lustre: 25186:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 770.508294] Lustre: 25186:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 770.516288] Lustre: 25186:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 770.529667] Lustre: 25186:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 770.538198] Lustre: 25186:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 770.553447] Lustre: 25186:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 770.572178] Lustre: 25186:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 770.582421] Lustre: 25186:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 770.588566] Lustre: 25186:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 770.595449] Lustre: 25186:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 770.602921] Lustre: 25186:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 770.608392] Lustre: 25186:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 779.009941] Lustre: 26209:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 779.026766] Lustre: 26209:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 779.031153] Lustre: 26209:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 779.042472] Lustre: 26209:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 779.051569] Lustre: 26209:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 779.066692] Lustre: 26209:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 779.083754] Lustre: 26209:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 779.092675] Lustre: 26209:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 779.101357] Lustre: 26209:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 779.108623] Lustre: 26209:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 779.121940] Lustre: 26209:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 779.136833] Lustre: 26209:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 790.759445] Lustre: 26209:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 790.774817] Lustre: 26209:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 790.782802] Lustre: 26209:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 790.789581] Lustre: 26209:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 790.795306] Lustre: 26209:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 790.799795] Lustre: 26209:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 790.803619] Lustre: 26209:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 790.807984] Lustre: 26209:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 790.812326] Lustre: 26209:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 790.816835] Lustre: 26209:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 790.820877] Lustre: 26209:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 790.829947] Lustre: 26209:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 836.074961] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 836.081509] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 836.102392] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 841.192888] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 842.311855] Lustre: server umount lustre-OST07c6 complete [ 854.166581] Lustre: server umount lustre-MDT0000 complete [ 874.289589] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 878.475631] Key type lgssc unregistered [ 878.961815] LNet: 28756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 878.973090] LNetError: 28756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 878.993104] LNet: Removed LNI 192.168.201.151@tcp [ 880.130985] Key type .llcrypt unregistered [ 880.134271] Key type ._llcrypt unregistered [ 900.547803] Key type ._llcrypt registered [ 900.551733] Key type .llcrypt registered [ 900.678750] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 901.999044] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 902.043642] alg: No test for adler32 (adler32-zlib) [ 903.236682] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 903.519491] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 905.248216] Key type lgssc registered [ 906.341861] Lustre: Echo OBD driver; http://www.lustre.org/ [ 914.295845] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 914.309953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 915.848823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 920.982035] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 925.164043] LustreError: 30028:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 925.187539] LustreError: 30028:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 929.877956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 930.183586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 934.610485] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 942.474944] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 949.797424] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 958.227524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 958.913777] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 960.964810] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 965.106150] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 973.357578] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 991.724352] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 991.730588] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 991.748037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 993.230752] Lustre: server umount lustre-OST0000 complete [ 1000.931170] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1000.935909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1000.946569] Lustre: Skipped 1 previous similar message [ 1000.966609] Lustre: Skipped 1 previous similar message [ 1003.901511] Lustre: server umount lustre-MDT0000 complete [ 1006.050168] LustreError: 30025:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1006.091310] LustreError: 30025:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1008.311637] LustreError: 30008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787115999 with bad export cookie 18149324578211511327 [ 1008.334550] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1019.098469] Lustre: server umount lustre-MDT0001 complete [ 1031.657078] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 1034.220454] Key type lgssc unregistered [ 1034.500744] LNet: 33766:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1034.514791] LNetError: 33766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1034.532391] LNet: Removed LNI 192.168.201.151@tcp [ 1035.316244] Key type .llcrypt unregistered [ 1035.323603] Key type ._llcrypt unregistered [ 1055.565550] Key type ._llcrypt registered [ 1055.570781] Key type .llcrypt registered [ 1055.679192] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 1056.756527] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1056.874031] alg: No test for adler32 (adler32-zlib) [ 1058.087438] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 1058.384106] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 1060.104520] Key type lgssc registered [ 1061.698267] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1069.546736] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1069.571096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1071.172394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1075.713910] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 1081.315371] LustreError: 35037:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1081.338950] LustreError: 35037:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1084.128606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1084.633066] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1089.836526] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 1098.569453] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1105.547062] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1113.169899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1113.806330] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1116.866768] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1120.669793] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 1128.835861] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1142.755607] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1142.763428] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1142.784049] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1144.808101] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1144.808262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1144.812554] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1144.836251] Lustre: Skipped 1 previous similar message [ 1147.290425] Lustre: server umount lustre-OST0000 complete [ 1151.464949] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1151.474608] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1151.504343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1155.042716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1155.050329] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1157.865961] Lustre: server umount lustre-MDT0000 complete [ 1160.168884] LustreError: 35038:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1160.193945] LustreError: 35038:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1162.884822] LustreError: 35017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787116153 with bad export cookie 2339916326366056659 [ 1162.892118] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1162.903715] LustreError: 35017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1163.469841] Lustre: server umount lustre-MDT0001 complete [ 1177.749708] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 1181.867142] Key type lgssc unregistered [ 1182.379197] LNet: 38586:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1182.395831] LNetError: 38586:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1182.428137] LNet: Removed LNI 192.168.201.151@tcp [ 1183.863162] Key type .llcrypt unregistered [ 1183.865691] Key type ._llcrypt unregistered [ 1216.999426] Key type ._llcrypt registered [ 1217.001867] Key type .llcrypt registered [ 1217.121249] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 1218.811958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1218.865277] alg: No test for adler32 (adler32-zlib) [ 1219.948316] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 1220.241357] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 1221.960782] Key type lgssc registered [ 1223.586779] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1233.390631] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1233.425524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1235.086053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1240.326475] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 1240.545212] LustreError: 40001:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1240.561102] LustreError: 40001:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1245.681624] LustreError: 40002:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1250.708640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1250.812235] LustreError: 40001:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1251.296447] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1256.409423] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 1264.853545] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1271.041645] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1277.992117] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1278.364059] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1280.433100] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1285.668628] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 1296.385033] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1303.896548] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.51@tcp (stopping) [ 1307.107758] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1307.116326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1307.130072] Lustre: Skipped 1 previous similar message [ 1307.150134] Lustre: Skipped 2 previous similar messages [ 1308.970277] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.51@tcp (stopping) [ 1309.770659] Lustre: server umount lustre-MDT0000 complete [ 1312.229344] LustreError: 40985:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1312.260042] LustreError: 40985:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1314.208461] LustreError: 39982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787116305 with bad export cookie 4814979281588992253 [ 1314.216547] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1314.219224] LustreError: 39982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1317.346406] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1317.348880] LustreError: 39997:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1317.361979] Lustre: Skipped 1 previous similar message [ 1317.375910] LustreError: 39997:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1317.400673] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1320.961847] Lustre: server umount lustre-MDT0001 complete [ 1352.160133] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1352.390594] Lustre: server umount lustre-OST0000 complete [ 1370.029814] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 1373.441887] Key type lgssc unregistered [ 1373.817829] LNet: 43867:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1373.822829] LNetError: 43867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1374.907547] LNet: Removed LNI 192.168.201.151@tcp [ 1376.064687] Key type .llcrypt unregistered [ 1376.067880] Key type ._llcrypt unregistered [ 1401.182896] Key type ._llcrypt registered [ 1401.188039] Key type .llcrypt registered [ 1401.323355] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 1402.283651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1402.448945] alg: No test for adler32 (adler32-zlib) [ 1403.508285] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 1403.730942] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 1405.424238] Key type lgssc registered [ 1406.934585] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1415.346665] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1415.364202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1416.860845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1421.089666] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 1426.414501] LustreError: 45284:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1426.458440] LustreError: 45284:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1430.029747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1430.492635] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1435.305675] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 1444.691191] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1451.312871] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1458.468600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1458.901801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1462.012617] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1465.501915] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 1473.883859] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1481.697688] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1481.708948] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1481.714550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1481.725496] Lustre: Skipped 1 previous similar message [ 1481.739729] Lustre: Skipped 1 previous similar message [ 1486.818776] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1486.829754] Lustre: Skipped 2 previous similar messages [ 1491.958133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1491.973316] Lustre: Skipped 2 previous similar messages [ 1495.008214] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1495.205096] Lustre: server umount lustre-OST0000 complete [ 1517.538226] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1517.543824] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1517.560265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1517.571045] Lustre: Skipped 2 previous similar messages [ 1519.358975] Lustre: server umount lustre-MDT0000 complete [ 1522.660090] LustreError: 45284:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1522.696183] LustreError: 45284:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1523.122531] LustreError: 45266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787116514 with bad export cookie 203731422487373578 [ 1523.132462] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1523.140074] LustreError: 45266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1529.615862] Lustre: server umount lustre-MDT0001 complete [ 1542.041837] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 1545.002619] Key type lgssc unregistered [ 1545.268651] LNet: 48994:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1545.278176] LNetError: 48994:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1545.288841] LNet: Removed LNI 192.168.201.151@tcp [ 1546.211498] Key type .llcrypt unregistered [ 1546.214133] Key type ._llcrypt unregistered [ 1574.282942] Key type ._llcrypt registered [ 1574.291626] Key type .llcrypt registered [ 1574.401293] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 1575.693994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1575.903069] alg: No test for adler32 (adler32-zlib) [ 1577.239321] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 1577.651881] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 1579.336194] Key type lgssc registered [ 1580.550773] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1589.567910] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1589.582891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1591.085531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1596.794125] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 1600.483025] LustreError: 50445:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1600.505717] LustreError: 50445:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1604.964094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1605.272626] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1609.609441] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 1617.780384] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1625.073345] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1632.522731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1632.852681] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1635.970978] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1639.837843] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 1647.190410] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1656.816342] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1660.524819] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1672.507376] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-19 1h17m42s [ 1674.135913] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1677.281705] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1677.290948] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1677.303903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1678.371266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1678.389604] Lustre: Skipped 4 previous similar messages [ 1680.709589] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.51@tcp (stopping) [ 1682.833984] Lustre: server umount lustre-MDT0000 complete [ 1684.468897] LustreError: 50445:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1686.431685] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1687.852224] LustreError: 52425:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1687.874714] LustreError: 52425:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1690.605088] LustreError: 51173:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1690.630844] LustreError: 51173:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1693.116681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1693.310043] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1693.557739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1693.624358] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1694.571169] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1699.124372] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 1703.307958] Lustre: DEBUG MARKER: Wait for df (36733) ... [ 1705.096913] Lustre: DEBUG MARKER: done [ 1712.099628] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1712.121298] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1712.157726] Lustre: Skipped 2 previous similar messages [ 1712.173190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1712.180791] Lustre: Skipped 1 previous similar message [ 1714.148299] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1714.160851] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1716.787989] Lustre: server umount lustre-OST0000 complete [ 1725.411987] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1725.414262] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1725.417315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1725.417323] Lustre: Skipped 3 previous similar messages [ 1725.456713] Lustre: Skipped 1 previous similar message [ 1735.136171] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1735.397803] Lustre: server umount lustre-MDT0000 complete [ 1738.722923] LustreError: 50446:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1738.753167] LustreError: 50446:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1738.884315] LustreError: 50426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787116729 with bad export cookie 17599277240327158097 [ 1738.884769] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1738.903983] LustreError: 50426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1739.267467] Lustre: server umount lustre-MDT0001 complete [ 1750.401439] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 1753.028490] Key type lgssc unregistered [ 1753.372178] LNet: 55371:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1753.387412] LNetError: 55371:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1753.404807] LNet: Removed LNI 192.168.201.151@tcp [ 1754.350612] Key type .llcrypt unregistered [ 1754.355193] Key type ._llcrypt unregistered [ 1790.082313] Key type ._llcrypt registered [ 1790.084777] Key type .llcrypt registered [ 1790.168764] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 1791.496380] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1791.519118] alg: No test for adler32 (adler32-zlib) [ 1792.629636] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 1792.874267] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 1794.568193] Key type lgssc registered [ 1795.853740] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1802.551782] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1802.571087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1803.776768] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1803.801851] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1804.025351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1807.769704] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 1816.126863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1816.219211] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1816.245968] Lustre: 57577:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1816.444639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1820.892826] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 1827.592478] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1833.355639] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1839.927701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1840.092614] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1840.233455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1846.299454] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 1848.980618] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1855.724523] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1862.069474] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-19 1h20m51s [ 1864.060836] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1871.454659] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1874.255592] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1874.345666] LustreError: 57583:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1894.746871] Lustre: lustre-MDT0000: Client 9882ce9c-f584-4d29-92c6-4d03cc679df4 (at 192.168.201.51@tcp) reconnecting [ 1914.361072] LustreError: 57583:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1917.387723] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1920.513156] Lustre: DEBUG MARKER: done [ 1930.726023] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1930.730426] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1930.755866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1931.234340] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1931.263625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1931.271236] Lustre: Skipped 1 previous similar message [ 1933.203298] Lustre: server umount lustre-OST0000 complete [ 1939.425641] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1939.429317] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1939.445121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1942.570224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1942.578647] Lustre: Skipped 4 previous similar messages [ 1944.309622] Lustre: server umount lustre-MDT0000 complete [ 1949.709242] LustreError: 57569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787116940 with bad export cookie 9120594012969998530 [ 1949.733635] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1949.734266] LustreError: 57569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1950.217099] Lustre: server umount lustre-MDT0001 complete [ 1963.271799] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 1965.662935] Key type lgssc unregistered [ 1965.989479] LNet: 61905:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1966.005301] LNetError: 61905:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1966.023236] LNet: Removed LNI 192.168.201.151@tcp [ 1966.793606] Key type .llcrypt unregistered [ 1966.798274] Key type ._llcrypt unregistered [ 2014.253581] Key type ._llcrypt registered [ 2014.256467] Key type .llcrypt registered [ 2014.497727] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 2016.511048] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2016.630027] alg: No test for adler32 (adler32-zlib) [ 2017.895809] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 2018.444798] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 2020.248297] Key type lgssc registered [ 2022.279745] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2026.944366] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2026.981283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2028.522578] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2028.576467] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2028.866687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2030.422636] Lustre: Failing over lustre-MDT0000 [ 2030.743449] Lustre: server umount lustre-MDT0000 complete [ 2038.624442] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 01:23:49 (1787117029) [ 2046.133246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2046.638206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2051.091778] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 2059.921178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2060.024386] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2060.064907] Lustre: 64709:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2060.539666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2066.466803] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 2076.987623] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2084.519169] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2091.184812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2091.420864] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2091.676673] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2097.825642] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2098.557577] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 2106.208134] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2112.091617] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2122.225957] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2122.241865] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2122.270114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2123.746027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2123.756409] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2125.384891] Lustre: server umount lustre-MDT0000 complete [ 2129.589266] LustreError: 64699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787117120 with bad export cookie 14338229469903145873 [ 2129.591550] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2129.603461] LustreError: 64699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2129.948910] Lustre: server umount lustre-MDT0001 complete [ 2133.423616] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2138.517300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2147.056228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2149.344540] Lustre: 63628:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787117124/real 1787117124] req@000000006d0733d1 x1873928004871296/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787117140 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2149.361385] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2149.369248] Lustre: Skipped 1 previous similar message [ 2150.376308] Lustre: 63630:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787117125/real 1787117125] req@000000005635db73 x1873928004871424/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787117141 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2153.442947] Lustre: 63631:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787117128/real 1787117128] req@00000000a08db8aa x1873928004871552/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787117144 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2154.773551] LustreError: 66556:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2154.784200] LustreError: 66556:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2154.856250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2154.968104] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2155.813677] Lustre: 63628:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787117130/real 1787117130] req@00000000c5186b36 x1873928004871808/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787117146 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2158.884827] LustreError: 66557:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2159.470941] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 2159.907159] LustreError: 66556:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2159.921193] LustreError: 66556:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2161.136084] Lustre: 63628:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787117135/real 1787117135] req@000000007bafca75 x1873928004871936/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787117151 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2165.222088] LustreError: 68476:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2165.260505] LustreError: 68476:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2169.306793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2169.814745] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2174.961812] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2175.611410] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 2178.105407] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2185.732945] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2194.197960] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2203.621306] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2203.635208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2203.651926] Lustre: Skipped 4 previous similar messages [ 2208.754247] LustreError: 68475:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2208.819916] Lustre: server umount lustre-MDT0000 complete [ 2213.695963] LustreError: 64701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787117204 with bad export cookie 14338229469903148148 [ 2213.711811] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2213.713269] LustreError: 64701:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2213.860961] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2213.875938] Lustre: Skipped 2 previous similar messages [ 2214.136736] Lustre: server umount lustre-MDT0001 complete [ 2221.915836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2233.046495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2238.433394] LustreError: 63627:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000c1ba3d30 x1873928004902272/t0(0) o250->MGC192.168.201.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2238.787405] LustreError: 69428:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2238.808939] LustreError: 69428:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2238.867254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2238.946875] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2244.022570] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2244.030392] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 2253.038729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2258.048929] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 2258.925411] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2266.663706] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2274.984876] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2284.516631] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2284.521828] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2284.523069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2284.575715] Lustre: Skipped 2 previous similar messages [ 2289.448905] Lustre: server umount lustre-MDT0000 complete [ 2289.635736] LustreError: 71162:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2289.651366] LustreError: 71162:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 2293.497808] LustreError: 64699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787117284 with bad export cookie 14338229469903149527 [ 2293.503075] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2293.509079] LustreError: 64699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2293.880163] Lustre: server umount lustre-MDT0001 complete [ 2300.298123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2303.572954] Lustre: DEBUG MARKER: files compared the same [ 2315.441487] Lustre: server umount lustre-OST0000 complete [ 2334.555441] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 2337.346623] Key type lgssc unregistered [ 2337.682860] LNet: 74441:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2337.687926] LNetError: 74441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2337.702888] LNet: Removed LNI 192.168.201.151@tcp [ 2338.668219] Key type .llcrypt unregistered [ 2338.670205] Key type ._llcrypt unregistered [ 2366.701914] Key type ._llcrypt registered [ 2366.704547] Key type .llcrypt registered [ 2366.790749] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 2367.835626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2368.017609] alg: No test for adler32 (adler32-zlib) [ 2369.206479] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 2369.599399] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 2371.272482] Key type lgssc registered [ 2372.627099] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2382.315968] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2382.355060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2383.855171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2388.724904] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug +malloc all [ 2393.065465] LustreError: 75860:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2393.101313] LustreError: 75860:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2398.178070] LustreError: 75859:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2398.258312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2398.503677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2402.383678] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug +malloc all [ 2410.376698] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2415.492603] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2421.777695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2422.010502] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2428.183254] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2429.141318] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug +malloc all [ 2438.412327] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2448.865573] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2448.869944] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2448.871526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2448.885258] Lustre: Skipped 1 previous similar message [ 2453.987652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2453.991662] Lustre: Skipped 1 previous similar message [ 2454.245556] Lustre: server umount lustre-OST0000 complete [ 2459.108339] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2459.115425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2459.143761] Lustre: Skipped 1 previous similar message [ 2459.152453] Lustre: Skipped 1 previous similar message [ 2464.225170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2464.234649] Lustre: Skipped 2 previous similar messages [ 2465.167515] Lustre: server umount lustre-MDT0000 complete [ 2469.344936] LustreError: 75854:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2469.388473] LustreError: 75854:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2469.622720] LustreError: 75842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787117460 with bad export cookie 11657433830188641415 [ 2469.641325] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2470.122766] Lustre: server umount lustre-MDT0001 complete [ 2483.011825] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 2485.728613] Key type lgssc unregistered [ 2485.992744] LNet: 79391:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2485.999745] LNetError: 79391:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2487.030474] LNet: Removed LNI 192.168.201.151@tcp [ 2488.266595] Key type .llcrypt unregistered [ 2488.279725] Key type ._llcrypt unregistered [ 2516.236155] Key type ._llcrypt registered [ 2516.240373] Key type .llcrypt registered [ 2516.358229] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 2518.013414] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2518.064419] alg: No test for adler32 (adler32-zlib) [ 2519.114900] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 2519.294082] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 2520.920157] Key type lgssc registered [ 2521.940832] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2529.372734] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2529.389324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2544.928413] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2550.048432] LustreError: 80784:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.151@tcp: failed processing log, type 4: rc = -110 [ 2576.736515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2582.991648] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 2590.373377] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2598.661281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2598.797937] Lustre: *** cfs_fail_loc=706, val=0*** [ 2599.284423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2599.389479] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2604.522531] LustreError: 80810:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2604.543776] LustreError: 80810:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2604.621935] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 2609.643283] LustreError: 80809:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2612.709984] LustreError: 81868:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2612.728500] LustreError: 81868:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2615.360663] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2615.752948] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2620.994391] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 2628.839217] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2635.259611] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2640.354977] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2640.360546] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2640.370341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2641.380676] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2641.387282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2641.393297] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2641.416927] Lustre: Skipped 1 previous similar message [ 2643.818795] Lustre: server umount lustre-OST0000 complete [ 2651.619128] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2651.629322] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2651.642759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2651.650570] Lustre: Skipped 1 previous similar message [ 2656.737816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2656.747393] Lustre: Skipped 1 previous similar message [ 2661.858711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2661.870989] Lustre: Skipped 1 previous similar message [ 2662.368204] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2662.581710] Lustre: server umount lustre-MDT0000 complete [ 2666.606821] LustreError: 80790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787117657 with bad export cookie 3268131102157812869 [ 2666.608605] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2666.622603] LustreError: 80790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2667.115376] Lustre: server umount lustre-MDT0001 complete [ 2680.727665] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 2684.111527] Key type lgssc unregistered [ 2684.550755] LNet: 84402:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2684.570092] LNetError: 84402:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2684.603557] LNet: Removed LNI 192.168.201.151@tcp [ 2685.907097] Key type .llcrypt unregistered [ 2685.908841] Key type ._llcrypt unregistered [ 2712.408510] Key type ._llcrypt registered [ 2712.413461] Key type .llcrypt registered [ 2712.537503] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 2713.689444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2713.899721] alg: No test for adler32 (adler32-zlib) [ 2714.951693] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 2715.179505] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 2716.857720] Key type lgssc registered [ 2718.105225] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2726.387418] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2726.412569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2733.615294] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 2744.040535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2744.876948] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2749.925610] LustreError: 86508:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2749.947206] LustreError: 86508:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2751.802664] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 2755.043387] LustreError: 86511:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2760.165476] LustreError: 86508:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2760.188713] LustreError: 86508:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2763.234318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2763.832941] LustreError: 86507:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2763.870196] LustreError: 86507:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2763.886684] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2768.876884] LustreError: 86506:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2768.888423] LustreError: 86506:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2771.514299] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 2780.422648] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2780.449070] Lustre: MGS: Client eb6dce42-3315-403b-9d84-5137500c5092 (at 0@lo) reconnecting [ 2780.465581] LustreError: 86522:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2780.717424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2780.804946] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2785.900122] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 2802.660979] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2802.677540] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2802.706889] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2804.860803] Lustre: server umount lustre-OST0000 complete [ 2811.366532] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2811.372118] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2811.399987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2811.405520] Lustre: Skipped 1 previous similar message [ 2811.420064] Lustre: Skipped 1 previous similar message [ 2816.481195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2816.496440] Lustre: Skipped 1 previous similar message [ 2819.571724] Lustre: server umount lustre-MDT0000 complete [ 2821.601192] LustreError: 86511:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2821.624597] LustreError: 86511:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 2824.923719] Lustre: server umount lustre-MDT0001 complete [ 2829.963049] Lustre: server umount MGS complete [ 2841.870176] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 01:37:11 (1787117831) [ 2874.463105] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [ 2885.023304] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 2920.335529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2926.280934] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 2938.249283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2938.639351] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2938.696467] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2938.777747] Lustre: lustre-MDT0001: new disk, initializing [ 2938.871970] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2938.891194] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2938.904679] Lustre: 93707:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2938.916344] LustreError: 93707:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2944.398248] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 2955.346332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2955.625796] Lustre: lustre-OST0000: new disk, initializing [ 2955.633479] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2955.645944] Lustre: 94473:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2955.749711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2962.924549] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 2965.666632] Lustre: 94921:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000076887732 x1873928735729408/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787117972 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2973.035337] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2980.772923] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2980.785338] Lustre: MGS: Client 0cf542fa-8ed0-48a5-86ac-ef58659327bc (at 0@lo) reconnecting [ 2980.815991] LustreError: 93693:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2980.858283] Lustre: 92974:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2980.909265] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2981.007254] Lustre: lustre-MDT0000: new disk, initializing [ 2981.163937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2981.187635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2981.205152] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2982.257212] Lustre: 95542:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000009c33b5fb x1873928735736448/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787117989 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2985.756313] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 2987.371838] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2991.338470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2991.350018] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2991.450503] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3017.190618] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3017.211810] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3017.224727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3019.238138] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3019.253020] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3022.380691] Lustre: server umount lustre-OST0000 complete [ 3025.889588] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3025.892890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3025.897152] Lustre: Skipped 1 previous similar message [ 3025.900827] Lustre: Skipped 3 previous similar messages [ 3031.968037] Lustre: server umount lustre-MDT0000 complete [ 3041.818779] Lustre: server umount lustre-MDT0001 complete [ 3045.391885] Lustre: server umount MGS complete [ 3056.409765] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 01:40:46 (1787118046) [ 3074.826710] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 3077.800739] Key type lgssc unregistered [ 3078.083195] LNet: 98049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3078.101882] LNetError: 98049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3078.131170] LNet: Removed LNI 192.168.201.151@tcp [ 3078.970756] Key type .llcrypt unregistered [ 3078.973526] Key type ._llcrypt unregistered [ 3100.323816] Key type ._llcrypt registered [ 3100.326723] Key type .llcrypt registered [ 3100.465577] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 3101.234315] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3101.333049] alg: No test for adler32 (adler32-zlib) [ 3102.387722] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 3102.797565] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 3104.408180] Key type lgssc registered [ 3105.893347] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3116.242934] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3116.281665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3118.242154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3123.944882] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 3128.809131] LustreError: 99271:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3128.850710] LustreError: 99271:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3133.926310] LustreError: 99272:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3133.993401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3134.408658] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3139.060136] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 3147.422351] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3153.214794] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3161.316247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3161.857073] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3163.894499] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3168.413183] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 3176.695574] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3189.730545] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3189.741484] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3189.766124] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3190.763081] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3190.770398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3190.775745] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3190.800251] Lustre: Skipped 1 previous similar message [ 3195.910271] LustreError: 101097:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3195.951810] LustreError: 101097:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3196.233431] Lustre: server umount lustre-OST0000 complete [ 3200.993184] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3201.007815] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3201.017177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3201.068952] Lustre: Skipped 1 previous similar message [ 3206.133052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3206.144844] Lustre: Skipped 2 previous similar messages [ 3206.897802] Lustre: server umount lustre-MDT0000 complete [ 3210.942973] LustreError: 99250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787118201 with bad export cookie 17374949928095246454 [ 3210.948215] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3210.965772] LustreError: 99250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3211.278796] Lustre: server umount lustre-MDT0001 complete [ 3222.459364] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 3225.182772] Key type lgssc unregistered [ 3225.437242] LNet: 102827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3225.442676] LNetError: 102827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3225.465351] LNet: Removed LNI 192.168.201.151@tcp [ 3226.435696] Key type .llcrypt unregistered [ 3226.440846] Key type ._llcrypt unregistered [ 3242.860453] Key type ._llcrypt registered [ 3242.862556] Key type .llcrypt registered [ 3242.942407] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 3244.324605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3244.402195] alg: No test for adler32 (adler32-zlib) [ 3245.497212] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 3245.712080] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 3247.400210] Key type lgssc registered [ 3248.724583] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3254.913812] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3254.927326] LustreError: 103913:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3254.943370] LustreError: 103916:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3254.952579] LustreError: 103913:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4983 [ 3254.960932] LustreError: 103913:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3254.973393] LustreError: 103913:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3254.982149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3256.622356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3264.066975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3264.401608] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3268.960891] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 3274.915858] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing lsmod [ 3277.813473] LustreError: 105106:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3277.883821] LustreError: 105121:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3277.890015] LustreError: 105106:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4930 [ 3277.899498] LustreError: 105106:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3277.907945] LustreError: 105106:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3277.926812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3278.331504] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3283.430760] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3283.444344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3283.445257] Lustre: Skipped 1 previous similar message [ 3283.452704] Lustre: Skipped 2 previous similar messages [ 3287.675094] Lustre: server umount lustre-MDT0000 complete [ 3288.547710] LustreError: 105158:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3288.566329] LustreError: 105158:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3289.573672] LustreError: 103953:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3289.602707] LustreError: 103953:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3291.947920] LustreError: 103933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787118282 with bad export cookie 16781093430372525110 [ 3291.956313] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3291.963653] LustreError: 103933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3291.979547] Lustre: 103946:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.151@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3293.795166] LustreError: 104384:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000fe1a4c4b x1873929291451264/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3294.022504] Lustre: server umount lustre-MDT0001 complete [ 3309.807151] Lustre: server umount lustre-OST0000 complete [ 3319.001859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3319.505411] LustreError: 106186:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3319.578829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3324.192894] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 3324.907265] LustreError: 106187:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3330.021633] LustreError: 106186:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3333.430119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3333.693682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3338.553250] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 3347.192878] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3352.699494] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3359.773233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3360.114120] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3364.219423] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3366.887146] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 3374.221425] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3388.901274] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3388.906199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3388.906455] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3388.914903] Lustre: Skipped 2 previous similar messages [ 3388.940641] Lustre: Skipped 1 previous similar message [ 3393.951912] Lustre: server umount lustre-OST0000 complete [ 3399.141234] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3399.179170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3399.195244] Lustre: Skipped 1 previous similar message [ 3404.262042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3404.274817] Lustre: Skipped 1 previous similar message [ 3404.865313] Lustre: server umount lustre-MDT0000 complete [ 3408.356956] LustreError: 106181:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3408.389436] LustreError: 106181:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3409.177442] LustreError: 106167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787118400 with bad export cookie 16781093430372525796 [ 3409.198244] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3409.545950] Lustre: server umount lustre-MDT0001 complete [ 3420.383214] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 3423.267699] Key type lgssc unregistered [ 3423.554185] LNet: 109740:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3423.573689] LNetError: 109740:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3423.590598] LNet: Removed LNI 192.168.201.151@tcp [ 3424.508533] Key type .llcrypt unregistered [ 3424.510265] Key type ._llcrypt unregistered [ 3454.829286] Key type ._llcrypt registered [ 3454.832968] Key type .llcrypt registered [ 3454.915604] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 3455.934071] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3456.005507] alg: No test for adler32 (adler32-zlib) [ 3457.430895] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 3457.845125] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 3459.560209] Key type lgssc registered [ 3461.118907] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3470.084450] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3470.099344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3471.704115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3475.761295] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 3481.069502] LustreError: 111156:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3481.076920] LustreError: 111156:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3484.607393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3485.146625] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3492.147983] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 3501.720548] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3508.591366] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3515.480148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3515.954966] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3516.992707] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3522.977133] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 3531.227601] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3538.045064] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3542.580534] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3547.620749] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3547.631703] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3547.650336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3549.665448] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3549.685079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3549.691094] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3549.731138] Lustre: Skipped 1 previous similar message [ 3553.618640] Lustre: server umount lustre-OST0000 complete [ 3559.911680] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3559.916734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3559.926080] Lustre: Skipped 1 previous similar message [ 3559.934891] Lustre: Skipped 1 previous similar message [ 3565.038732] LustreError: 113598:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3565.058237] LustreError: 113598:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3565.093150] Lustre: server umount lustre-MDT0000 complete [ 3569.139633] LustreError: 111157:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3569.183652] LustreError: 111157:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3570.128707] LustreError: 111136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787118561 with bad export cookie 1172265366265609631 [ 3570.135862] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3570.146758] LustreError: 111136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3570.796450] Lustre: server umount lustre-MDT0001 complete [ 3584.159973] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 3586.672614] Key type lgssc unregistered [ 3586.923842] LNet: 114799:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3586.930457] LNetError: 114799:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3586.940231] LNet: Removed LNI 192.168.201.151@tcp [ 3587.662144] Key type .llcrypt unregistered [ 3587.664181] Key type ._llcrypt unregistered [ 3606.359848] Key type ._llcrypt registered [ 3606.361949] Key type .llcrypt registered [ 3606.455236] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 3607.898756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3608.016974] alg: No test for adler32 (adler32-zlib) [ 3609.469625] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 3609.953486] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 3611.769249] Key type lgssc registered [ 3613.749527] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3625.272197] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3625.284466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3627.047865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3633.401523] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 3636.205164] LustreError: 116021:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3636.229975] LustreError: 116021:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3641.318236] LustreError: 116022:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3643.891124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3644.535432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3649.357180] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 3657.645638] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3664.483363] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3672.837726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3673.193319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3680.573533] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 3690.373538] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3696.774338] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3712.482236] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3712.489977] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3712.496674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3712.506318] Lustre: Skipped 1 previous similar message [ 3712.517692] Lustre: Skipped 1 previous similar message [ 3717.101854] Lustre: server umount lustre-OST0000 complete [ 3721.200511] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3721.215206] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3721.241866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3722.724694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3722.737314] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3727.496120] Lustre: server umount lustre-MDT0000 complete [ 3727.855180] LustreError: 116022:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3727.876963] LustreError: 116022:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3732.440404] LustreError: 116002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787118723 with bad export cookie 12480814609996690322 [ 3732.441968] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3732.458277] LustreError: 116002:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3732.918888] Lustre: server umount lustre-MDT0001 complete [ 3745.940983] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 3749.151483] Key type lgssc unregistered [ 3749.560098] LNet: 119580:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3749.567619] LNetError: 119580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3749.612398] LNet: Removed LNI 192.168.201.151@tcp [ 3750.639266] Key type .llcrypt unregistered [ 3750.643909] Key type ._llcrypt unregistered [ 3779.923751] Key type ._llcrypt registered [ 3779.926031] Key type .llcrypt registered [ 3780.091349] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 3781.384131] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3781.597887] alg: No test for adler32 (adler32-zlib) [ 3782.650178] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 3782.861723] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 3784.456238] Key type lgssc registered [ 3785.909616] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3794.583906] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3794.607102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3796.173536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3801.020642] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 3806.695662] LustreError: 121001:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3806.722897] LustreError: 121001:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3809.380343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3809.835036] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3814.425842] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 3821.624840] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3828.525830] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3835.714173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3836.178015] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3842.376236] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3843.398401] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 3852.567214] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3862.441699] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3866.780647] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3866.787793] Lustre: Skipped 2 previous similar messages [ 3871.509613] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3871.514661] Lustre: Skipped 2 previous similar messages [ 3872.393642] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3872.429931] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3877.456915] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.51@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3877.462655] Lustre: Skipped 2 previous similar messages [ 3878.448219] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.51@tcp 192.168.0.[2,10]@tcp [ 3884.922243] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.51/24@tcp in log lustre-MDT0000 [ 3884.934263] Lustre: Skipped 2 previous similar messages [ 3888.116706] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.51/24@tcp [ 3888.120228] Lustre: Skipped 1 previous similar message [ 3895.793434] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3895.794089] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3895.808637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3895.811070] Lustre: Skipped 1 previous similar message [ 3896.701477] Lustre: server umount lustre-OST0000 complete [ 3900.900848] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3900.913085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3900.923410] Lustre: Skipped 2 previous similar messages [ 3906.029685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3906.034244] Lustre: Skipped 2 previous similar messages [ 3906.729789] Lustre: server umount lustre-MDT0000 complete [ 3910.301644] LustreError: 120982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787118901 with bad export cookie 3699662395634598456 [ 3910.321198] LustreError: 120982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3910.321464] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3910.623739] Lustre: server umount lustre-MDT0001 complete [ 3922.077326] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 3925.021301] Key type lgssc unregistered [ 3925.314634] LNet: 125696:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3925.317275] LNetError: 125696:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3925.359112] LNet: Removed LNI 192.168.201.151@tcp [ 3926.152528] Key type .llcrypt unregistered [ 3926.157671] Key type ._llcrypt unregistered [ 3959.716557] Key type ._llcrypt registered [ 3959.725106] Key type .llcrypt registered [ 3959.902110] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 3962.065924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3962.144770] alg: No test for adler32 (adler32-zlib) [ 3963.581971] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 3963.932853] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 3965.607125] Key type lgssc registered [ 3967.045480] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3977.326647] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3977.337030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3978.965354] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3979.018043] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.51/24@tcp [ 3979.034564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3984.399580] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 3988.454063] LustreError: 127262:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3988.466520] LustreError: 127262:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3993.368267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3993.572665] LustreError: 127261:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3993.848986] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3993.877797] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.51/24@tcp [ 3993.947612] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3998.777502] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 4007.492182] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4014.200491] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4022.414382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4022.840930] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4023.878173] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 4028.927788] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 4036.839897] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4046.830295] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4046.838121] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4046.853153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4048.355088] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4048.360241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4048.372765] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4048.395106] Lustre: Skipped 1 previous similar message [ 4052.361958] Lustre: server umount lustre-OST0000 complete [ 4058.599025] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4058.604832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4058.611609] Lustre: Skipped 1 previous similar message [ 4063.716573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4063.728242] Lustre: Skipped 2 previous similar messages [ 4068.835241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4068.847786] Lustre: Skipped 1 previous similar message [ 4070.368187] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4070.594600] Lustre: server umount lustre-MDT0000 complete [ 4073.713061] LustreError: 127240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787119064 with bad export cookie 11124350998888666462 [ 4073.727454] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4073.728366] LustreError: 127240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4074.065176] Lustre: server umount lustre-MDT0001 complete [ 4084.877820] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 4087.749261] Key type lgssc unregistered [ 4088.078467] LNet: 130851:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4088.092395] LNetError: 130851:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4088.109473] LNet: Removed LNI 192.168.201.151@tcp [ 4088.902608] Key type .llcrypt unregistered [ 4088.906314] Key type ._llcrypt unregistered [ 4115.235434] Key type ._llcrypt registered [ 4115.239396] Key type .llcrypt registered [ 4115.378136] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [ 4130.038422] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 4131.368363] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4131.447183] alg: No test for adler32 (adler32-zlib) [ 4132.550644] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 4132.832243] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 4134.520439] Key type lgssc registered [ 4135.763730] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4175.563812] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4175.605555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4176.881226] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4176.938182] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4177.057763] Lustre: lustre-MDT0000: new disk, initializing [ 4177.190289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4177.218056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4181.414657] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 4191.334036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4191.481473] Lustre: 134254:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4191.523258] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4191.529573] Lustre: Skipped 1 previous similar message [ 4191.606458] Lustre: lustre-MDT0001: new disk, initializing [ 4191.738917] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4191.789374] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4191.809033] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4196.382606] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 4206.772551] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4215.031915] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4224.902413] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4225.136707] Lustre: lustre-OST0000: new disk, initializing [ 4225.140814] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4225.147118] Lustre: 136220:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4225.228837] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4232.542527] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 4235.378459] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4235.393570] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4235.545422] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4244.169974] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4249.999773] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4250.323513] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4254.674338] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4254.910567] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4261.345085] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4261.353395] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4261.358882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4261.371115] Lustre: Skipped 1 previous similar message [ 4263.626824] Lustre: server umount lustre-OST0000 complete [ 4268.513702] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4268.521941] LustreError: Skipped 1 previous similar message [ 4268.524207] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4268.531860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4268.534904] Lustre: Skipped 1 previous similar message [ 4271.587958] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4271.589648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4271.609840] Lustre: Skipped 1 previous similar message [ 4274.508994] Lustre: server umount lustre-MDT0000 complete [ 4278.084836] LustreError: 134244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787119269 with bad export cookie 16981352199330989147 [ 4278.095642] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4278.097937] LustreError: 134244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4278.822778] Lustre: server umount lustre-MDT0001 complete [ 4287.539500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4288.056178] LustreError: 138453:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4288.071918] LustreError: 138453:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4288.130738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4292.604808] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 4293.602618] LustreError: 138454:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4298.727637] LustreError: 138454:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4301.289947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4301.689386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4306.263568] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 4313.794721] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4319.986670] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4327.134026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4327.514862] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4334.369917] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 4341.846850] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4347.167765] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4347.423595] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4353.347600] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4353.644046] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4363.924292] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4364.155237] Lustre: lustre-OST0001: new disk, initializing [ 4364.159994] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4364.166203] Lustre: 141822:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4364.259302] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4372.698849] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 4374.209797] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4374.237518] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4374.456511] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4382.513504] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4382.768625] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4387.739184] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4387.980486] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4394.543893] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d0e031cd000.ost_server_uuid 50 [ 4396.603561] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8d0e031cd000.ost_server_uuid in IDLE state after 0 sec [ 4405.219640] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4405.230664] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4405.255077] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4407.322262] Lustre: server umount lustre-OST0001 complete [ 4410.346299] LustreError: 143215:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4410.356225] LustreError: 143215:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4413.922583] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4413.940887] LustreError: Skipped 1 previous similar message [ 4413.959631] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4413.983933] Lustre: Skipped 1 previous similar message [ 4413.996773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4414.010903] Lustre: Skipped 1 previous similar message [ 4415.461294] LustreError: 140275:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4415.478338] LustreError: 140275:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4425.184180] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4425.352469] Lustre: server umount lustre-OST0000 complete [ 4429.795098] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4429.808360] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4429.824409] Lustre: Skipped 1 previous similar message [ 4429.840460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4429.852122] Lustre: Skipped 4 previous similar messages [ 4435.357397] Lustre: server umount lustre-MDT0000 complete [ 4435.944041] LustreError: 138448:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4435.960853] LustreError: 138448:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4439.458551] LustreError: 138435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787119430 with bad export cookie 16981352199330990022 [ 4439.461480] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4439.473958] LustreError: 138435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4439.779188] Lustre: server umount lustre-MDT0001 complete [ 4461.154458] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 4464.472707] Key type lgssc unregistered [ 4464.728579] LNet: 144829:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4464.733394] LNetError: 144829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4464.770953] LNet: Removed LNI 192.168.201.151@tcp [ 4465.715876] Key type .llcrypt unregistered [ 4465.718237] Key type ._llcrypt unregistered [ 4512.508288] Key type ._llcrypt registered [ 4512.510649] Key type .llcrypt registered [ 4512.672152] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [ 4526.881925] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 4528.274808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4528.302871] alg: No test for adler32 (adler32-zlib) [ 4529.334753] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 4529.601664] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 4531.272435] Key type lgssc registered [ 4532.233758] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4569.093847] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4569.120133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4570.395503] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4570.413700] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4570.505531] Lustre: lustre-MDT0000: new disk, initializing [ 4570.641217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4570.744309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4575.958955] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 4586.048288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4586.188484] Lustre: 148985:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4586.258613] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4586.264340] Lustre: Skipped 1 previous similar message [ 4586.339543] Lustre: lustre-MDT0001: new disk, initializing [ 4586.438444] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4586.477780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4586.490166] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4590.333907] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 4600.060929] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4605.542205] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4612.497778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4612.945682] Lustre: lustre-OST0000: new disk, initializing [ 4612.961388] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4612.971319] Lustre: 150954:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4613.107114] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4619.626686] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 4621.884312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4621.893950] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4622.013374] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4630.534656] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4635.774058] Lustre: Failing over lustre-OST0000 [ 4636.118523] Lustre: server umount lustre-OST0000 complete [ 4654.053585] Lustre: 147211:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787119628/real 1787119628] req@00000000d8f6a840 x1873930637837952/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787119644 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4654.053964] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4654.076893] Lustre: 147211:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 4654.106128] Lustre: Skipped 1 previous similar message [ 4655.434482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4655.715152] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4655.736194] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4658.784168] Lustre: 147210:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787119633/real 1787119633] req@0000000093d1ba87 x1873930637839104/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787119649 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4661.895112] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 4664.288378] Lustre: 147210:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787119638/real 1787119638] req@000000002c56f9c1 x1873930637840128/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787119654 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4664.305931] Lustre: 147210:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4664.881304] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4665.199643] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4665.202235] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4666.270860] Lustre: Failing over lustre-MDT0000 [ 4667.001196] Lustre: server umount lustre-MDT0000 complete [ 4668.320290] Lustre: 147210:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787119643/real 1787119643] req@000000008c88dc68 x1873930637841152/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787119659 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4668.354408] Lustre: 147210:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4668.385117] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4668.394373] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4669.922640] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4669.959436] Lustre: Skipped 1 previous similar message [ 4675.044010] LustreError: 148996:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4675.087287] LustreError: 148996:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 4680.169457] LustreError: 148993:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4680.221206] LustreError: 148993:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4685.298905] LustreError: 148996:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4685.330970] LustreError: 148996:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4685.782469] Lustre: 147210:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787119660/real 1787119660] req@000000000757fcbf x1873930637851136/t0(0) o400->MGC192.168.201.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787119676 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4685.827386] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4687.218665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4690.735978] LustreError: 148991:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4690.743558] LustreError: 148991:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4695.849390] LustreError: 151674:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4695.879498] LustreError: 151674:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4696.339389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4696.398822] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4700.971924] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 4700.976230] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4701.680797] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4701.692456] Lustre: Skipped 1 previous similar message [ 4701.717263] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4701.815052] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4711.406943] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4711.418652] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4711.440477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4715.469139] Lustre: server umount lustre-OST0000 complete [ 4722.148354] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4722.148946] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4722.152939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4722.152949] Lustre: Skipped 1 previous similar message [ 4722.204378] Lustre: Skipped 2 previous similar messages [ 4725.996045] Lustre: server umount lustre-MDT0000 complete [ 4727.271920] LustreError: 148996:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4727.283318] LustreError: 148996:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4729.665406] LustreError: 148977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787119720 with bad export cookie 6354889902613268899 [ 4729.668618] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4729.681334] LustreError: 148977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4729.971633] Lustre: server umount lustre-MDT0001 complete [ 4741.895851] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 4744.362991] Key type lgssc unregistered [ 4744.712114] LNet: 154610:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4744.720713] LNetError: 154610:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4745.768637] LNet: Removed LNI 192.168.201.151@tcp [ 4746.979452] Key type .llcrypt unregistered [ 4746.981647] Key type ._llcrypt unregistered [ 4786.021229] Key type ._llcrypt registered [ 4786.031196] Key type .llcrypt registered [ 4786.204231] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 4787.119135] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4787.196079] alg: No test for adler32 (adler32-zlib) [ 4788.246919] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 4788.447035] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 4790.096205] Key type lgssc registered [ 4791.347671] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4799.029769] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4799.088670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4800.749996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4805.450833] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 4810.214192] LustreError: 156148:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4810.231235] LustreError: 156148:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4814.712760] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4815.335038] LustreError: 156149:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4818.518205] LustreError: 156177:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4818.535610] LustreError: 156177:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4818.548284] LustreError: 156177:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 4819.025763] Lustre: server umount lustre-MDT0000 complete [ 4834.994831] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [ 4845.978323] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 4886.508410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4886.750674] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4886.789838] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4886.893594] Lustre: lustre-MDT0000: new disk, initializing [ 4887.019234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4887.039974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4892.152719] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 4903.527050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4903.672257] Lustre: 159483:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4903.702199] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4903.705029] Lustre: Skipped 1 previous similar message [ 4903.818368] Lustre: lustre-MDT0001: new disk, initializing [ 4903.957391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4904.004973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4904.033869] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4908.236983] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 4917.832930] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4925.333913] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4933.083599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4933.301368] Lustre: lustre-OST0000: new disk, initializing [ 4933.305843] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4933.312661] Lustre: 161452:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4933.410362] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4939.376514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4939.386473] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4939.544834] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4939.828542] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 4948.591799] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4953.202655] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4955.224731] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 02:12:25 (1787119945) [ 5175.264874] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5175.269207] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5175.270425] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5175.274025] Lustre: Skipped 1 previous similar message [ 5178.990891] Lustre: server umount lustre-MDT0000 complete [ 5180.393451] LustreError: 159492:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5180.408544] LustreError: 159492:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5182.481445] LustreError: 159475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787120173 with bad export cookie 4345498944075107659 [ 5182.487137] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5182.494262] LustreError: 159475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5185.511424] LustreError: 159491:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5185.517135] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5185.595985] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5185.602415] Lustre: Skipped 2 previous similar messages [ 5187.553364] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5189.153207] Lustre: server umount lustre-MDT0001 complete [ 5199.363587] Lustre: server umount lustre-OST0000 complete [ 5209.656920] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [ 5221.188143] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 5258.471406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5258.809776] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5258.857636] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5258.957921] Lustre: lustre-MDT0000: new disk, initializing [ 5259.078615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5259.113075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5263.655505] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 5274.181980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5274.313860] Lustre: 166110:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5274.365738] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5274.369194] Lustre: Skipped 1 previous similar message [ 5274.424734] Lustre: lustre-MDT0001: new disk, initializing [ 5274.500900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5274.530168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5274.540481] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5278.585152] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 5288.563358] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5294.245441] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5300.442140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5300.707985] Lustre: lustre-OST0000: new disk, initializing [ 5300.715304] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5300.726396] Lustre: 168079:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5300.834176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5307.490603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5307.506562] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5307.523865] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 5307.658617] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5317.128545] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5321.868585] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5322.100882] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5326.297722] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5326.532179] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5331.437732] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5331.448642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5331.463200] Lustre: Skipped 1 previous similar message [ 5331.487388] Lustre: Skipped 1 previous similar message [ 5336.547732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5336.555658] Lustre: Skipped 1 previous similar message [ 5343.200113] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5343.482913] Lustre: server umount lustre-OST0000 complete [ 5351.393681] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5351.397794] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5351.414513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5351.417178] Lustre: Skipped 2 previous similar messages [ 5353.713892] Lustre: server umount lustre-MDT0000 complete [ 5357.036205] LustreError: 166116:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5357.051099] LustreError: 166116:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5357.814322] LustreError: 168078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787120348 with bad export cookie 4345498944075266489 [ 5357.817514] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5357.826940] LustreError: 168078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5358.324730] Lustre: server umount lustre-MDT0001 complete [ 5367.678205] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:19:17 (1787120357) [ 5386.319384] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [ 5395.345336] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 5429.197435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5429.471807] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5429.517076] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5429.675212] Lustre: lustre-MDT0000: new disk, initializing [ 5429.883907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5429.899831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5435.669322] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 5446.223523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5446.342610] Lustre: 172835:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5446.547606] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5451.281598] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 5460.724071] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5466.064304] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5473.789479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5474.162516] Lustre: lustre-OST0000: new disk, initializing [ 5474.165166] Lustre: Skipped 1 previous similar message [ 5474.176949] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5474.187603] Lustre: Skipped 2 previous similar messages [ 5474.193062] Lustre: 174813:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5480.652899] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 5483.058747] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5483.069040] Lustre: Skipped 1 previous similar message [ 5483.078559] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5483.161433] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5490.629495] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5498.849491] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5498.852205] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5498.859060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5498.871050] Lustre: Skipped 2 previous similar messages [ 5498.887898] Lustre: Skipped 2 previous similar messages [ 5503.553024] Lustre: server umount lustre-OST0000 complete [ 5508.065644] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5508.075067] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5521.376450] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5521.739788] Lustre: server umount lustre-MDT0000 complete [ 5523.425891] LustreError: 172844:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5523.441101] LustreError: 172844:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5525.409445] LustreError: 172828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787120516 with bad export cookie 4345498944075267364 [ 5525.416975] LustreError: 172828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5525.417298] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5525.972663] Lustre: server umount lustre-MDT0001 complete [ 5534.210911] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:22:04 (1787120524) [ 5550.379947] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [ 5559.560978] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 5600.081404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5600.506575] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5600.574526] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5600.639283] Lustre: lustre-MDT0000: new disk, initializing [ 5600.700279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5600.703426] Lustre: Skipped 2 previous similar messages [ 5600.714616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5605.778695] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 5618.003073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5618.293941] Lustre: 179140:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5618.782724] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5623.939284] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 5636.427582] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5644.120110] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5651.653936] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5652.152076] Lustre: 181119:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5659.870956] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 5661.790341] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5661.968834] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5672.870681] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5682.659125] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5682.666895] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5682.671917] LustreError: Skipped 1 previous similar message [ 5682.672751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5682.672756] Lustre: Skipped 7 previous similar messages [ 5682.709684] Lustre: Skipped 2 previous similar messages [ 5688.080917] Lustre: server umount lustre-OST0000 complete [ 5698.965370] Lustre: server umount lustre-MDT0000 complete [ 5702.597827] LustreError: 181117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787120693 with bad export cookie 4345498944075268456 [ 5702.600424] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5702.605863] LustreError: 181117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5702.939455] Lustre: server umount lustre-MDT0001 complete [ 5715.294735] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 5718.873328] Key type lgssc unregistered [ 5719.322673] LNet: 182978:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5719.361506] LNetError: 182978:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5719.370893] LNet: Removed LNI 192.168.201.151@tcp [ 5720.331669] Key type .llcrypt unregistered [ 5720.334084] Key type ._llcrypt unregistered [ 5749.427959] Key type ._llcrypt registered [ 5749.431990] Key type .llcrypt registered [ 5749.544737] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 5751.104441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5751.181202] alg: No test for adler32 (adler32-zlib) [ 5752.348194] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 5752.699080] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 5754.458198] Key type lgssc registered [ 5756.138435] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5764.348915] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5764.379284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5765.870512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5769.655853] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 5775.340384] LustreError: 184395:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5775.362481] LustreError: 184395:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5778.185851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5778.634196] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5783.432349] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 5791.172802] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5797.278740] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5804.088510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5804.547547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5810.406936] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 5810.691606] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5817.741787] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5839.332896] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5839.378846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5839.385490] Lustre: Skipped 1 previous similar message [ 5844.450876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5845.148386] Lustre: server umount lustre-OST0000 complete [ 5849.572174] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5849.592649] Lustre: Skipped 1 previous similar message [ 5849.601883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5849.616359] Lustre: Skipped 2 previous similar messages [ 5854.690802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5854.697128] Lustre: Skipped 1 previous similar message [ 5859.811674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5859.818729] Lustre: Skipped 1 previous similar message [ 5863.392729] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5863.561715] Lustre: server umount lustre-MDT0000 complete [ 5864.932531] LustreError: 184395:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5864.947799] LustreError: 184395:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5867.762605] LustreError: 185131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787120858 with bad export cookie 11466676056698027349 [ 5867.767809] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5867.775932] LustreError: 185131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5868.152652] Lustre: server umount lustre-MDT0001 complete [ 5878.804450] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 5881.542558] Key type lgssc unregistered [ 5881.924677] LNet: 187952:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5881.929097] LNetError: 187952:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5881.949865] LNet: Removed LNI 192.168.201.151@tcp [ 5882.846564] Key type .llcrypt unregistered [ 5882.847752] Key type ._llcrypt unregistered [ 5908.182602] Key type ._llcrypt registered [ 5908.186718] Key type .llcrypt registered [ 5908.326617] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 5909.719445] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5909.751072] alg: No test for adler32 (adler32-zlib) [ 5910.739483] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 5910.939874] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 5912.592243] Key type lgssc registered [ 5913.863400] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5922.012276] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5922.043378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5923.525653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5928.205402] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 5934.058348] LustreError: 189371:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5934.076731] LustreError: 189371:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5937.346551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5937.803903] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5942.567671] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 5950.367454] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5955.810573] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5962.257809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5962.614270] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5969.799466] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5970.131149] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 5978.661986] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5985.256630] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5985.260314] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5985.266647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5985.284520] Lustre: Skipped 1 previous similar message [ 5986.640768] Lustre: lustre-OST0000: Not available for connect from 192.168.201.51@tcp (stopping) [ 5986.653785] Lustre: Skipped 1 previous similar message [ 5990.370676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5995.490147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5995.501201] Lustre: Skipped 2 previous similar messages [ 5999.072696] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5999.284555] Lustre: server umount lustre-OST0000 complete [ 6006.058631] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8d0e08d39800.ost_server_uuid 50 [ 6007.717146] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6024.677681] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6024.684594] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6024.697049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6024.701639] Lustre: Skipped 2 previous similar messages [ 6025.779102] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6026.741452] Lustre: server umount lustre-MDT0000 complete [ 6030.822949] LustreError: 189371:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6030.839574] LustreError: 189371:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6031.353699] LustreError: 189352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787121022 with bad export cookie 568865742209050924 [ 6031.367714] LustreError: 189352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6031.368024] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6031.782982] Lustre: server umount lustre-MDT0001 complete [ 6041.095958] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 02:30:30 (1787121030) [ 6048.152417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6048.662658] LustreError: 193284:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6048.782351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6053.311772] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 6053.866699] LustreError: 193283:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6058.984554] LustreError: 193284:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6061.786469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6062.241782] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6066.445907] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 6074.405295] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6079.731291] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6087.945941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6088.269561] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6093.423697] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6094.391209] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 6101.443507] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6107.318263] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6107.501542] Lustre: lustre-OST0001: new disk, initializing [ 6107.513932] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6107.521611] Lustre: 196125:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6107.611988] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6113.128339] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 6113.349873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6113.353991] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6113.441932] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6122.537966] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6128.473732] Lustre: lustre-OST0000: Not available for connect from 192.168.201.51@tcp (stopping) [ 6128.479321] Lustre: Skipped 2 previous similar messages [ 6129.129429] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6129.138786] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6139.361602] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6139.496444] Lustre: server umount lustre-OST0000 complete [ 6143.807273] LustreError: 195104:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6145.703876] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6145.987361] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6150.732103] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6150.906517] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6159.849147] LustreError: 195104:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6159.870955] LustreError: 195104:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 6164.975677] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6164.978299] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6164.991336] LustreError: Skipped 1 previous similar message [ 6164.997817] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6165.007061] Lustre: Skipped 2 previous similar messages [ 6165.039073] Lustre: Skipped 9 previous similar messages [ 6170.351800] Lustre: server umount lustre-OST0001 complete [ 6175.218865] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6175.218951] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6175.248624] Lustre: Skipped 1 previous similar message [ 6181.011959] Lustre: server umount lustre-MDT0000 complete [ 6185.118639] LustreError: 193265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787121176 with bad export cookie 568865742209051995 [ 6185.121158] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6185.128913] LustreError: 193265:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6185.599722] Lustre: server umount lustre-MDT0001 complete [ 6213.481174] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 02:33:23 (1787121203) [ 6222.913398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6223.386017] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6223.408263] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6223.646383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6227.956644] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 6237.435462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6237.575697] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6237.646409] Lustre: 199569:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6242.749862] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 6250.848368] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6256.144340] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6263.306647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6263.623120] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6270.109388] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6270.262126] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 6277.451209] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6284.754640] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6284.870982] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6293.069643] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 6300.776573] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6304.251359] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6304.270179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6304.277600] Lustre: Skipped 7 previous similar messages [ 6309.887576] Lustre: server umount lustre-OST0000 complete [ 6311.725358] LustreError: 203076:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6311.747361] LustreError: 203076:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6329.832120] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6329.845856] LustreError: Skipped 1 previous similar message [ 6332.971612] Lustre: server umount lustre-OST0001 complete [ 6340.072120] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6340.088553] Lustre: Skipped 3 previous similar messages [ 6344.105247] Lustre: server umount lustre-MDT0000 complete [ 6347.735814] LustreError: 201437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787121338 with bad export cookie 568865742209053402 [ 6347.737728] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6347.744062] LustreError: 201437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6348.023735] Lustre: server umount lustre-MDT0001 complete [ 6373.834986] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 02:36:03 (1787121363) [ 6395.101436] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [ 6405.653273] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 6440.299789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6440.497981] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6440.520649] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6440.589880] Lustre: lustre-MDT0000: new disk, initializing [ 6440.673554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6440.682461] Lustre: Skipped 3 previous similar messages [ 6440.696452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6445.132828] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 6454.951272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6455.060207] Lustre: 207665:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6455.115418] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6455.119384] Lustre: Skipped 1 previous similar message [ 6455.223375] Lustre: lustre-MDT0001: new disk, initializing [ 6455.373927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6455.388395] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6459.159419] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 6469.698995] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6476.131509] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6485.011149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6485.485698] Lustre: lustre-OST0000: new disk, initializing [ 6485.496051] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6485.513351] Lustre: 209633:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6492.701943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6492.717602] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6492.796546] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6493.108943] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 6503.747561] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6509.935560] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6510.270503] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6514.640220] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6515.080618] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6518.256762] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6518.257914] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6518.263733] LustreError: Skipped 1 previous similar message [ 6518.272851] Lustre: Skipped 1 previous similar message [ 6518.275978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6518.282279] Lustre: Skipped 8 previous similar messages [ 6523.370972] LustreError: 209618:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6523.390895] LustreError: 209618:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 6523.495262] Lustre: server umount lustre-OST0000 complete [ 6533.969548] Lustre: server umount lustre-MDT0000 complete [ 6538.568028] LustreError: 207658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787121529 with bad export cookie 568865742209054963 [ 6538.577400] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6538.592857] LustreError: 207658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6538.950299] Lustre: server umount lustre-MDT0001 complete [ 6549.743853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6556.033092] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 6565.941539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6571.141976] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 6579.097572] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6585.865801] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6593.972731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6594.397532] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6594.403506] Lustre: Skipped 4 previous similar messages [ 6600.937660] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 6608.013917] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6613.111554] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6613.418808] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6619.716521] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6619.976055] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6622.190733] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6628.220147] Lustre: server umount lustre-OST0000 complete [ 6634.421906] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6639.524328] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6668.825618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6676.534957] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 6677.551540] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6677.575760] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6677.575775] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6684.892832] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6686.599883] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6692.844547] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6692.864873] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6692.872020] Lustre: Skipped 5 previous similar messages [ 6692.883222] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6692.883233] Lustre: Skipped 8 previous similar messages [ 6697.363565] Lustre: server umount lustre-OST0000 complete [ 6707.403220] Lustre: server umount lustre-MDT0000 complete [ 6708.198273] LustreError: 211847:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6708.240964] LustreError: 211847:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 6711.628047] LustreError: 211833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787121702 with bad export cookie 568865742209055845 [ 6711.638229] LustreError: 211833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6711.654746] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6712.126374] Lustre: server umount lustre-MDT0001 complete [ 6721.897284] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 02:41:52 (1787121712) [ 6728.032610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6734.130310] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 6743.175316] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6748.175280] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 6756.516187] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6762.961760] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6771.733768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6779.784441] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 6788.987747] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6794.761715] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6794.927142] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6799.863602] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6800.150297] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6807.769558] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6808.137732] Lustre: lustre-OST0001: new disk, initializing [ 6808.157881] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6808.179756] Lustre: 221190:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6815.425242] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 6815.871627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6815.896942] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6816.008860] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6825.409636] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6830.912394] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6831.160662] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6835.992528] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6836.188572] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6841.833542] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6844.237236] Lustre: server umount lustre-OST0001 complete [ 6850.596678] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6850.909328] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6855.730783] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6856.035133] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6885.642076] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6885.835270] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6885.842856] Lustre: Skipped 5 previous similar messages [ 6891.019249] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6891.032361] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6891.047419] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6891.071541] Lustre: Skipped 1 previous similar message [ 6893.081138] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 6901.416279] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6902.865527] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6911.482263] Lustre: server umount lustre-OST0001 complete [ 6942.177159] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6942.447586] Lustre: server umount lustre-OST0000 complete [ 6952.425116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6952.432030] Lustre: Skipped 16 previous similar messages [ 6952.627486] Lustre: server umount lustre-MDT0000 complete [ 6956.715975] LustreError: 218554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787121947 with bad export cookie 568865742209056811 [ 6956.729682] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6956.743993] LustreError: 218554:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6971.361281] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6971.710760] Lustre: server umount lustre-MDT0001 complete [ 6994.519962] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 02:46:24 (1787121984) [ 7001.405391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7001.703477] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7001.763068] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7006.465459] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 7015.429447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7015.569786] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7015.627964] Lustre: 226437:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7021.244083] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 7028.978316] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7034.473598] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7040.999093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7041.146167] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7048.017775] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 7057.361215] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7067.653152] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7067.746327] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7074.032076] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 7081.866964] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7083.478094] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7088.527050] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7088.760128] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7093.523430] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7093.737235] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7100.793421] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d0e0377c800.ost_server_uuid 50 [ 7102.619981] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8d0e0377c800.ost_server_uuid in IDLE state after 0 sec [ 7103.627111] Lustre: Permanently deactivating lustre-OST0001 [ 7103.644599] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7108.865951] Lustre: Permanently reactivating lustre-OST0001 [ 7108.883646] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7108.889541] Lustre: Skipped 2 previous similar messages [ 7113.193732] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7113.204874] LustreError: Skipped 5 previous similar messages [ 7113.209630] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7113.237571] Lustre: Skipped 10 previous similar messages [ 7119.418477] Lustre: server umount lustre-OST0001 complete [ 7123.950108] LustreError: 228286:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7123.978750] LustreError: 228286:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 40 previous similar messages [ 7138.272429] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7138.628938] Lustre: server umount lustre-OST0000 complete [ 7148.915403] Lustre: server umount lustre-MDT0000 complete [ 7153.144829] LustreError: 226432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787122144 with bad export cookie 568865742209058099 [ 7153.148624] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7153.157727] LustreError: 226432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7153.518360] Lustre: server umount lustre-MDT0001 complete [ 7181.731416] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 02:49:31 (1787122171) [ 7191.033248] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7191.456816] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7191.494629] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7191.501322] Lustre: Skipped 2 previous similar messages [ 7197.088071] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 7206.056843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7206.180942] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7206.215785] Lustre: 233138:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7210.412909] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 7217.868735] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7223.657661] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7230.424246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7230.645600] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7230.674174] Lustre: Permanently deactivating lustre-OST0000 [ 7230.693431] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7238.458603] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 7239.767510] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7248.758874] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7257.067461] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7257.165534] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7263.993150] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7264.068364] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 7271.094578] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7273.127961] Lustre: Permanently reactivating lustre-OST0000 [ 7273.142313] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7273.146644] Lustre: Skipped 2 previous similar messages [ 7273.438063] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7273.440082] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 34 seconds. I think it's dead, and I am evicting it. exp 00000000ec7e911c, cur 1787122264 deadline 1787122260 last 1787122230 [ 7273.445351] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7273.464716] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7273.521347] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7300.064385] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7300.175594] Lustre: server umount lustre-OST0001 complete [ 7310.167718] Lustre: server umount lustre-OST0000 complete [ 7319.972440] Lustre: server umount lustre-MDT0000 complete [ 7324.029024] LustreError: 233131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787122314 with bad export cookie 568865742209059765 [ 7324.037972] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7324.039797] LustreError: 233131:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7324.487667] Lustre: server umount lustre-MDT0001 complete [ 7337.006608] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 7339.827379] Key type lgssc unregistered [ 7340.197579] LNet: 238020:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7340.205248] LNetError: 238020:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7340.220978] LNet: Removed LNI 192.168.201.151@tcp [ 7340.959349] Key type .llcrypt unregistered [ 7340.961256] Key type ._llcrypt unregistered [ 7363.977584] Key type ._llcrypt registered [ 7363.988705] Key type .llcrypt registered [ 7364.129067] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 7365.368574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7365.395915] alg: No test for adler32 (adler32-zlib) [ 7366.741597] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 7367.114292] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 7368.872491] Key type lgssc registered [ 7370.355923] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7380.557588] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7380.579031] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7382.122085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7388.106179] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 7391.202540] LustreError: 239483:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7391.208896] LustreError: 239483:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7392.231875] LustreError: 239484:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7396.905463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7397.013553] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7397.051030] Lustre: 239471:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7397.057781] Lustre: Permanently deactivating lustre-MDT0001 [ 7397.067446] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7397.437668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7397.865195] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7397.873123] LustreError: 239514:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7397.882230] LustreError: 239514:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7397.890367] LustreError: 239514:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 1, failed: rc = -108 [ 7403.628704] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 7411.728785] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7417.891311] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7424.643200] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7424.941260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7427.000911] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7428.003906] LustreError: 241324:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7431.012377] LustreError: 241324:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7431.347256] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 7436.273102] LustreError: 241647:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7439.224631] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7442.432130] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7445.508753] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7445.780443] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7451.128767] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7451.133671] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7451.697378] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 7459.970691] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7462.548814] Lustre: Permanently reactivating lustre-MDT0001 [ 7462.565854] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7462.570130] Lustre: Skipped 2 previous similar messages [ 7468.380538] Lustre: lustre-MDT0001: Client 1459ef9e-854d-40ba-bf21-6c1cde73e7b0 (at 192.168.201.51@tcp) reconnecting [ 7476.510484] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7478.549774] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7485.276411] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7485.541788] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7490.819742] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7491.030587] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7493.424726] Lustre: Permanently deactivating lustre-MDT0001 [ 7493.458173] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7493.463101] Lustre: Skipped 2 previous similar messages [ 7499.468387] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7509.976580] Lustre: server umount lustre-MDT0000 complete [ 7510.021508] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7510.030253] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7510.063844] LustreError: 240800:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7510.088030] LustreError: 240800:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7512.558789] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7512.584500] Lustre: Skipped 2 previous similar messages [ 7515.012438] LustreError: 240223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787122505 with bad export cookie 8121464155137387573 [ 7515.014788] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7515.026925] LustreError: 240223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7515.439390] Lustre: server umount lustre-MDT0001 complete [ 7521.115387] Lustre: server umount lustre-OST0000 complete [ 7534.376199] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 02:55:24 (1787122524) [ 7548.394498] Lustre: server umount lustre-OST0001 complete [ 7555.591722] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [ 7568.142405] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 7604.301231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7604.619858] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7604.635291] Lustre: Skipped 2 previous similar messages [ 7604.681368] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7604.801321] Lustre: lustre-MDT0000: new disk, initializing [ 7604.865864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7604.883517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7609.285804] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 7621.781494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7621.982837] Lustre: 247742:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7622.132932] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7622.136722] Lustre: Skipped 1 previous similar message [ 7622.233384] Lustre: lustre-MDT0001: new disk, initializing [ 7622.371911] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7622.445101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7622.461100] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7627.904698] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 7638.711595] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7644.912422] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7652.564860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7652.963250] Lustre: lustre-OST0000: new disk, initializing [ 7652.976042] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7652.989780] Lustre: 249713:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7653.111714] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7659.758285] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 7660.661843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7660.670823] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7660.832898] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7668.981624] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7670.889530] LustreError: 247749:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7679.766832] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7679.971558] Lustre: lustre-OST0001: new disk, initializing [ 7679.976199] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7679.981254] Lustre: 250826:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7680.975896] LustreError: 247749:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7681.025804] LustreError: 250122:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7686.221801] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 7689.807272] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7689.823873] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7689.930783] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7691.128197] LustreError: 250122:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7695.524878] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7697.156331] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7700.320824] Lustre: lustre-OST0001: Not available for connect from 192.168.201.51@tcp (stopping) [ 7700.452809] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7700.457457] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7700.503690] LustreError: Skipped 1 previous similar message [ 7705.408293] Lustre: lustre-OST0001: Not available for connect from 192.168.201.51@tcp (stopping) [ 7705.423708] Lustre: Skipped 2 previous similar messages [ 7710.517953] Lustre: lustre-OST0001: Not available for connect from 192.168.201.51@tcp (stopping) [ 7710.521522] Lustre: Skipped 2 previous similar messages [ 7713.250133] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7713.601297] Lustre: server umount lustre-OST0001 complete [ 7715.648557] LustreError: 249696:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7715.699798] LustreError: 249696:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 7736.312756] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7736.329865] Lustre: Skipped 1 previous similar message [ 7736.336522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7736.342101] Lustre: Skipped 2 previous similar messages [ 7739.925322] Lustre: server umount lustre-OST0000 complete [ 7745.517796] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7745.530387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7745.551927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7749.894496] Lustre: server umount lustre-MDT0000 complete [ 7751.661083] LustreError: 247755:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7751.674312] LustreError: 247755:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 7753.600587] LustreError: 247736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787122744 with bad export cookie 8121464155137390387 [ 7753.614197] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7753.626238] LustreError: 247736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7759.937441] Lustre: server umount lustre-MDT0001 complete [ 7769.908394] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 7772.413496] Key type lgssc unregistered [ 7772.643331] LNet: 252862:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7772.650169] LNetError: 252862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7772.665526] LNet: Removed LNI 192.168.201.151@tcp [ 7773.416666] Key type .llcrypt unregistered [ 7773.420684] Key type ._llcrypt unregistered [ 7812.508044] Key type ._llcrypt registered [ 7812.510341] Key type .llcrypt registered [ 7812.657074] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 7813.637379] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7813.894030] alg: No test for adler32 (adler32-zlib) [ 7815.012777] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 7815.286176] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 7817.016280] Key type lgssc registered [ 7818.248450] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7826.256646] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7826.272409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7827.503354] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7827.523983] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7827.748780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7831.838490] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 7840.484412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7840.580812] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7840.621124] Lustre: 255069:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7840.941919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7845.632972] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 7853.687543] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7860.181573] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7867.844331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7868.081606] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7868.339786] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7874.726635] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7876.585988] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 7885.716103] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7941.093421] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7941.119096] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7941.137408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7941.620687] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7941.641524] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7941.666909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7946.181158] Lustre: server umount lustre-OST0000 complete [ 7950.833945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7963.808871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7964.264483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7972.379950] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 7979.438406] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7981.620554] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7981.630311] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7981.632652] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7981.663716] Lustre: Skipped 1 previous similar message [ 8022.506395] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8022.519481] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8022.531474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8022.545561] Lustre: Skipped 1 previous similar message [ 8022.565359] Lustre: Skipped 1 previous similar message [ 8026.837581] Lustre: server umount lustre-OST0000 complete [ 8032.738446] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8032.765733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8037.796299] Lustre: server umount lustre-MDT0000 complete [ 8042.049549] LustreError: 255811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787123033 with bad export cookie 9124885225864777221 [ 8042.061572] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8042.071741] LustreError: 255811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8042.335808] Lustre: server umount lustre-MDT0001 complete [ 8054.103210] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 8058.100672] Key type lgssc unregistered [ 8058.541955] LNet: 260506:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8058.554136] LNetError: 260506:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8058.592908] LNet: Removed LNI 192.168.201.151@tcp [ 8060.134524] Key type .llcrypt unregistered [ 8060.149366] Key type ._llcrypt unregistered [ 8089.247446] Key type ._llcrypt registered [ 8089.249662] Key type .llcrypt registered [ 8089.509892] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 8091.231356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8091.954828] alg: No test for adler32 (adler32-zlib) [ 8092.950860] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 8093.145736] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 8094.816205] Key type lgssc registered [ 8096.271709] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8105.634113] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8105.674399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8107.551235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8113.221269] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 8117.752224] LustreError: 261925:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8117.791396] LustreError: 261925:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8122.852080] LustreError: 261926:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8123.720283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8124.037237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8129.352856] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 8137.576698] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8145.600400] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8153.767190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8154.009422] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8160.563592] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 8167.614547] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8177.594828] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8203.234238] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8203.257180] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8203.259453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8203.302819] Lustre: Skipped 1 previous similar message [ 8205.871633] Lustre: server umount lustre-OST0000 complete [ 8210.913532] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8210.922314] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8210.938570] Lustre: Skipped 1 previous similar message [ 8210.944694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8213.474358] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8213.474887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8215.894115] Lustre: server umount lustre-MDT0000 complete [ 8218.594923] LustreError: 261926:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8218.608097] LustreError: 261926:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8220.113937] LustreError: 261904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787123211 with bad export cookie 17483829208229651372 [ 8220.128436] LustreError: 261904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8220.133083] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8220.401900] Lustre: server umount lustre-MDT0001 complete [ 8232.057731] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 8234.377347] Key type lgssc unregistered [ 8234.688887] LNet: 266251:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8234.694731] LNetError: 266251:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8234.719444] LNet: Removed LNI 192.168.201.151@tcp [ 8235.653588] Key type .llcrypt unregistered [ 8235.655293] Key type ._llcrypt unregistered [ 8253.826053] Key type ._llcrypt registered [ 8253.828394] Key type .llcrypt registered [ 8253.904943] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 8255.264973] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8255.289192] alg: No test for adler32 (adler32-zlib) [ 8256.395433] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 8256.700867] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 8258.385293] Key type lgssc registered [ 8259.259139] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8266.564826] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8266.593784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8268.100955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8273.198968] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 8278.504247] LustreError: 267476:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8278.523792] LustreError: 267476:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8283.527729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8283.617186] LustreError: 267477:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8283.839216] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8288.531323] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 8295.560115] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8300.805868] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8308.224787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8308.652989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8313.824967] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8315.251880] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 8322.174538] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8334.305228] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8334.317519] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8334.337404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8334.376618] Lustre: Skipped 1 previous similar message [ 8338.999591] Lustre: server umount lustre-OST0000 complete [ 8344.549081] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8344.576062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8344.587654] Lustre: Skipped 2 previous similar messages [ 8349.027162] Lustre: server umount lustre-MDT0000 complete [ 8349.693922] LustreError: 267472:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8349.724812] LustreError: 267472:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8353.398312] LustreError: 267457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787123344 with bad export cookie 1421538108067835316 [ 8353.407577] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8353.412127] LustreError: 267457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8353.685884] Lustre: server umount lustre-MDT0001 complete [ 8365.300700] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 8368.089024] Key type lgssc unregistered [ 8368.359598] LNet: 271153:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8368.369871] LNetError: 271153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8368.392611] LNet: Removed LNI 192.168.201.151@tcp [ 8369.296135] Key type .llcrypt unregistered [ 8369.298355] Key type ._llcrypt unregistered [ 8391.933732] Key type ._llcrypt registered [ 8391.936728] Key type .llcrypt registered [ 8392.027557] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 8393.199218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8393.253066] alg: No test for adler32 (adler32-zlib) [ 8394.336577] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 8394.642861] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 8396.360178] Key type lgssc registered [ 8397.532601] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8405.436833] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8405.454814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8406.813993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8411.989934] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 8416.294993] LustreError: 272571:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8416.310832] LustreError: 272571:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8420.926485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8421.137803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8425.610768] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 8434.502758] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8441.392963] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8449.508714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8450.096883] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8451.150588] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8457.550381] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 8465.633651] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8497.130488] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8497.131427] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8497.132994] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8497.154755] Lustre: Skipped 1 previous similar message [ 8497.163603] LustreError: Skipped 1 previous similar message [ 8498.733213] Lustre: server umount lustre-OST0000 complete [ 8503.266365] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8503.287972] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8503.312590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8503.317282] Lustre: Skipped 1 previous similar message [ 8507.362610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8507.371685] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8508.910978] Lustre: server umount lustre-MDT0000 complete [ 8511.864488] LustreError: 272552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787123502 with bad export cookie 15861077808225368246 [ 8511.873168] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8511.889219] LustreError: 272552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8512.243563] Lustre: server umount lustre-MDT0001 complete [ 8521.484625] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 8523.683404] Key type lgssc unregistered [ 8523.917898] LNet: 276877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8523.923224] LNetError: 276877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8524.966090] LNet: Removed LNI 192.168.201.151@tcp [ 8525.759982] Key type .llcrypt unregistered [ 8525.762824] Key type ._llcrypt unregistered [ 8540.219387] Key type ._llcrypt registered [ 8540.222500] Key type .llcrypt registered [ 8540.364735] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 8541.394560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8541.562651] alg: No test for adler32 (adler32-zlib) [ 8542.628327] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 8542.851217] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 8544.496135] Key type lgssc registered [ 8545.456204] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8552.024820] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8552.037350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8553.377727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8557.054492] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 8563.686945] LustreError: 278092:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8563.704415] LustreError: 278092:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8564.681483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8565.242534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8569.573193] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 8576.861701] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8582.620906] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8590.618468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8590.832543] LustreError: 279938:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8591.025688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8598.534285] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8599.270182] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 8608.735593] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8624.098464] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8624.108452] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8624.117067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8624.125465] Lustre: Skipped 1 previous similar message [ 8624.131665] Lustre: Skipped 1 previous similar message [ 8629.221752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8629.240382] Lustre: Skipped 1 previous similar message [ 8629.818071] Lustre: server umount lustre-OST0000 complete [ 8636.899172] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8636.910642] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8636.930156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8639.461676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8639.463310] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8644.577885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8644.597636] Lustre: Skipped 2 previous similar messages [ 8649.186141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8649.637508] Lustre: server umount lustre-MDT0000 complete [ 8649.705941] LustreError: 278087:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8649.739329] LustreError: 278087:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8654.847448] LustreError: 278103:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8654.892252] LustreError: 278103:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8655.019637] LustreError: 278057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787123645 with bad export cookie 18264569808385284845 [ 8655.024783] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8655.038406] LustreError: 278057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8655.635817] Lustre: server umount lustre-MDT0001 complete [ 8668.847911] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 8671.568498] Key type lgssc unregistered [ 8671.942537] LNet: 281796:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8671.951752] LNetError: 281796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8671.986624] LNet: Removed LNI 192.168.201.151@tcp [ 8672.943840] Key type .llcrypt unregistered [ 8672.951120] Key type ._llcrypt unregistered [ 8685.547623] Key type ._llcrypt registered [ 8685.551301] Key type .llcrypt registered [ 8685.646576] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8702.530811] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [ 8716.175983] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 8717.202336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8717.273138] alg: No test for adler32 (adler32-zlib) [ 8718.428680] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 8718.735259] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 8720.426554] Key type lgssc registered [ 8722.032397] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8754.536398] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8754.576795] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8755.868270] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8755.890908] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8755.988813] Lustre: lustre-MDT0000: new disk, initializing [ 8756.100357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8756.119418] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8760.903724] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 8771.767404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8771.902603] Lustre: 285348:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8771.950070] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8771.957904] Lustre: Skipped 1 previous similar message [ 8772.063380] Lustre: lustre-MDT0001: new disk, initializing [ 8772.232240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8772.287142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8772.303728] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8777.496845] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 8789.833509] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8798.052063] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8806.823797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8807.226651] Lustre: lustre-OST0000: new disk, initializing [ 8807.233101] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8807.262799] Lustre: 287319:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8807.459656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8813.139134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8813.157678] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8813.307454] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8814.414735] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 8822.577082] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8826.437340] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8826.614667] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8831.032713] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8831.164163] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8834.025123] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8834.027223] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8834.036879] LustreError: Skipped 1 previous similar message [ 8834.046941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8834.056291] Lustre: Skipped 1 previous similar message [ 8834.103522] Lustre: Skipped 1 previous similar message [ 8838.928588] Lustre: server umount lustre-OST0000 complete [ 8843.752738] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8843.757517] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8843.773787] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8850.511713] Lustre: server umount lustre-MDT0000 complete [ 8853.847320] LustreError: 285343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787123844 with bad export cookie 14588021893455092112 [ 8853.851895] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8853.855893] LustreError: 285343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8854.238874] Lustre: server umount lustre-MDT0001 complete [ 8863.895997] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 03:17:33 (1787123853) [ 8870.383586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8870.736755] LustreError: 289730:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8870.758202] LustreError: 289730:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8870.819962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8875.316692] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 8876.020247] LustreError: 289731:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8880.099598] LustreError: 289730:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8882.980491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8883.360539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8887.586987] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 8894.131324] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8899.556465] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8906.067822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8906.627881] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8912.775867] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 8919.566706] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8929.250930] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8929.260271] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8929.281539] Lustre: Skipped 1 previous similar message [ 8929.287274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8929.295875] Lustre: Skipped 2 previous similar messages [ 8931.308243] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8931.315794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8931.346951] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8931.368631] Lustre: Skipped 1 previous similar message [ 8933.929952] Lustre: server umount lustre-OST0000 complete [ 8939.490785] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8939.503449] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8939.529818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8944.816317] Lustre: server umount lustre-MDT0000 complete [ 8946.659101] LustreError: 289731:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8946.677554] LustreError: 289731:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8948.227343] LustreError: 289710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787123939 with bad export cookie 14588021893455092994 [ 8948.227833] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8948.247023] LustreError: 289710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8948.651238] Lustre: server umount lustre-MDT0001 complete [ 8960.253276] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 8962.886423] Key type lgssc unregistered [ 8963.255792] LNet: 293266:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8963.261391] LNetError: 293266:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8963.278814] LNet: Removed LNI 192.168.201.151@tcp [ 8964.067789] Key type .llcrypt unregistered [ 8964.073309] Key type ._llcrypt unregistered [ 8989.716455] Key type ._llcrypt registered [ 8989.720898] Key type .llcrypt registered [ 8989.837509] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [ 9002.187934] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 9003.052573] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9003.199401] alg: No test for adler32 (adler32-zlib) [ 9004.252850] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 9004.444874] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 9006.072237] Key type lgssc registered [ 9007.253759] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9049.890686] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9049.942210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9051.198247] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9051.232129] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9051.318185] Lustre: lustre-MDT0000: new disk, initializing [ 9051.399148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9051.408698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9056.039665] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 9066.862926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9066.984701] Lustre: 297076:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9067.054459] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9067.061023] Lustre: Skipped 1 previous similar message [ 9067.138563] Lustre: lustre-MDT0001: new disk, initializing [ 9067.256756] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9067.302582] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9067.321626] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9072.654673] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 9083.120678] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9089.304742] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9096.174030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9096.408353] Lustre: lustre-OST2710: new disk, initializing [ 9096.412304] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9096.419835] Lustre: 299044:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9096.512642] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9102.799806] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 9103.979779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9104.002604] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9104.196471] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9112.920725] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9119.904582] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9120.089608] Lustre: lustre-OST03e8: new disk, initializing [ 9120.097379] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9120.104810] Lustre: 300107:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9120.248553] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9125.862265] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 9129.586836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9129.593975] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9129.727082] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9132.601201] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9139.856788] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9140.065355] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9144.479607] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9144.648265] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9148.563354] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9148.744859] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9152.822105] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9153.014847] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9159.649367] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9159.663412] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9159.671701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9160.686945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9160.688661] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9160.699398] Lustre: Skipped 2 previous similar messages [ 9160.728841] Lustre: Skipped 2 previous similar messages [ 9162.979582] Lustre: server umount lustre-MDT0000 complete [ 9165.966073] LustreError: 299054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787124156 with bad export cookie 14626642161246439431 [ 9165.968936] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9165.974671] LustreError: 299054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9166.277461] Lustre: server umount lustre-MDT0001 complete [ 9179.188283] Lustre: server umount lustre-OST2710 complete [ 9193.070394] Lustre: server umount lustre-OST03e8 complete [ 9209.138634] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [ 9217.381886] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 9249.458607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9249.644297] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9249.666552] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9249.741210] Lustre: lustre-MDT0000: new disk, initializing [ 9249.811112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9249.822266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9253.595628] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 9262.891099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9262.953269] Lustre: 305359:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9262.990736] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9262.994100] Lustre: Skipped 1 previous similar message [ 9263.057306] Lustre: lustre-MDT0001: new disk, initializing [ 9263.127649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9263.170657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9263.180173] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9267.138806] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 9275.637663] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9280.972859] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9286.601146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9286.827964] Lustre: lustre-OST0000: new disk, initializing [ 9286.832965] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9286.840624] Lustre: 307326:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9286.950891] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9292.784150] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 9294.902986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9294.917665] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9295.000316] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9301.349307] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9305.472797] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9305.738632] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9309.658877] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9309.894847] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9315.813531] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9315.819287] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9315.827283] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9315.839013] Lustre: Skipped 1 previous similar message [ 9315.854986] Lustre: Skipped 2 previous similar messages [ 9317.712950] Lustre: server umount lustre-OST0000 complete [ 9320.929869] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9320.934218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9320.942849] Lustre: Skipped 1 previous similar message [ 9320.950823] Lustre: Skipped 1 previous similar message [ 9326.055526] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9326.073056] Lustre: Skipped 1 previous similar message [ 9335.264142] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9335.282515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9335.290625] Lustre: Skipped 3 previous similar messages [ 9335.844718] Lustre: server umount lustre-MDT0000 complete [ 9339.771456] LustreError: 305350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787124330 with bad export cookie 14626642161246441111 [ 9339.776469] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9339.783410] LustreError: 305350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9340.135487] Lustre: server umount lustre-MDT0001 complete [ 9349.093648] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 03:25:39 (1787124339) [ 9350.667299] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9352.649194] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 03:25:42 (1787124342) [ 9357.396341] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 9384.803252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9385.005935] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9385.034747] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9385.234613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9389.706293] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 9397.420298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9397.535106] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9397.582764] Lustre: 311185:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9402.523583] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 9423.343383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9423.588565] Lustre: Denying initial registration attempt from nid 192.168.201.151@tcp, specified as failover [ 9423.599883] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9423.609365] LustreError: 312636:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9423.617260] LustreError: 312636:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9423.621788] LustreError: 312636:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9423.728831] Lustre: server umount lustre-OST0000 complete [ 9423.732403] LustreError: 312636:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9432.036109] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9432.062196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9432.083373] Lustre: Skipped 1 previous similar message [ 9437.892279] Lustre: server umount lustre-MDT0000 complete [ 9441.774860] LustreError: 311177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787124432 with bad export cookie 14626642161246441986 [ 9441.793756] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9441.797306] LustreError: 311177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9442.149225] Lustre: server umount lustre-MDT0001 complete [ 9451.976691] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 9454.606790] Key type lgssc unregistered [ 9454.897796] LNet: 313760:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9454.908326] LNetError: 313760:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9454.931793] LNet: Removed LNI 192.168.201.151@tcp [ 9455.748190] Key type .llcrypt unregistered [ 9455.750350] Key type ._llcrypt unregistered [ 9466.431873] Key type ._llcrypt registered [ 9466.434302] Key type .llcrypt registered [ 9466.498816] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 9467.430856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9467.565335] alg: No test for adler32 (adler32-zlib) [ 9468.639406] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 9468.866270] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 9470.520648] Key type lgssc registered [ 9471.737464] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9519.582591] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 9528.281160] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9528.294974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9529.568430] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9529.598962] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9529.727663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9533.890527] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 9541.472339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9541.665870] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9541.735387] Lustre: 316524:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9542.037158] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9546.280487] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 9553.319408] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9558.638969] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9578.604471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9578.809150] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9579.034967] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9585.982859] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 9593.922640] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9596.899143] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9596.923063] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9596.947576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9597.432562] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9597.450171] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9597.464137] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9597.476763] Lustre: Skipped 1 previous similar message [ 9602.241977] Lustre: server umount lustre-OST0000 complete [ 9607.649511] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9607.678493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9612.769481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9612.779792] Lustre: Skipped 2 previous similar messages [ 9613.316336] Lustre: server umount lustre-MDT0000 complete [ 9616.870677] LustreError: 316517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787124607 with bad export cookie 6711000749036718992 [ 9616.880394] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9616.886392] LustreError: 316517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9617.194353] Lustre: server umount lustre-MDT0001 complete [ 9626.788985] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 9629.098445] Key type lgssc unregistered [ 9629.335262] LNet: 320102:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9629.344167] LNetError: 320102:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9629.357359] LNet: Removed LNI 192.168.201.151@tcp [ 9630.165629] Key type .llcrypt unregistered [ 9630.169358] Key type ._llcrypt unregistered [ 9659.355137] Key type ._llcrypt registered [ 9659.360157] Key type .llcrypt registered [ 9659.503944] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 9660.810626] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9660.877584] alg: No test for adler32 (adler32-zlib) [ 9662.044316] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 9662.323394] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 9664.112846] Key type lgssc registered [ 9665.196947] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9673.039786] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9673.061233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9674.503267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9679.426869] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 9684.961653] LustreError: 321520:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9684.970617] LustreError: 321520:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9687.633507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9687.956479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9692.586021] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 9700.717289] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9706.839718] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9715.061034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9715.462618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9721.854194] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 9730.298865] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9744.356627] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9744.363916] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9744.376655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9747.279579] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.51@tcp (stopping) [ 9748.454064] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9748.454762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9748.466837] Lustre: Skipped 1 previous similar message [ 9748.488331] Lustre: Skipped 2 previous similar messages [ 9749.904398] Lustre: server umount lustre-MDT0000 complete [ 9752.373225] LustreError: 321514:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9753.214316] LustreError: 321502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787124744 with bad export cookie 7782884735530895080 [ 9753.251775] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9753.569921] LustreError: 322763:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9753.571666] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9753.612751] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9757.480855] LustreError: 321516:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9758.692461] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9758.697443] Lustre: Skipped 1 previous similar message [ 9759.608118] Lustre: server umount lustre-MDT0001 complete [ 9765.529679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9774.141598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9778.537922] LustreError: 323355:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9778.556992] LustreError: 323355:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9778.624492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9778.703344] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9782.591530] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 9788.842791] LustreError: 323356:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9788.862566] LustreError: 323356:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9788.865045] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9790.431574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9790.793741] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9794.785786] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 9796.096705] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9801.271802] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9802.715852] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9807.153633] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9808.584799] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9814.497593] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9814.507383] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9814.530682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9816.547871] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9818.555455] Lustre: server umount lustre-OST0000 complete [ 9826.788955] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9826.818031] Lustre: Skipped 1 previous similar message [ 9828.687945] Lustre: server umount lustre-MDT0000 complete [ 9831.912044] LustreError: 325143:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9831.928413] LustreError: 325143:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9832.389305] LustreError: 324330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787124823 with bad export cookie 7782884735530902738 [ 9832.395791] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9832.408649] LustreError: 324330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9832.906796] Lustre: server umount lustre-MDT0001 complete [ 9842.963901] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [ 9845.505766] Key type lgssc unregistered [ 9845.762767] LNet: 327808:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9845.779289] LNetError: 327808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9846.822085] LNet: Removed LNI 192.168.201.151@tcp [ 9847.722329] Key type .llcrypt unregistered [ 9847.724227] Key type ._llcrypt unregistered [ 9867.116813] Key type ._llcrypt registered [ 9867.119328] Key type .llcrypt registered [ 9867.212442] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [ 9868.408978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9868.437386] alg: No test for adler32 (adler32-zlib) [ 9869.457408] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [ 9869.643732] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [ 9871.290080] Key type lgssc registered [ 9872.331981] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9879.328724] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9879.356701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9880.806727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9885.218716] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 9890.277162] LustreError: 329206:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9890.293101] LustreError: 329206:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9892.919919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9893.327266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9897.274990] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 9904.534621] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9910.300420] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9913.136991] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9913.150084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9913.150537] Lustre: Skipped 1 previous similar message [ 9913.165470] Lustre: Skipped 1 previous similar message [ 9918.308885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9918.319968] Lustre: Skipped 1 previous similar message [ 9919.503763] Lustre: server umount lustre-MDT0000 complete [ 9922.677655] LustreError: 329187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787124913 with bad export cookie 13555340349252155442 [ 9922.680176] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9922.689415] LustreError: 329187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9923.281413] Lustre: server umount lustre-MDT0001 complete [ 9937.561909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9938.092414] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9938.197093] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9938.557813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9944.961390] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 9957.422962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9957.551953] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9957.596737] Lustre: 331792:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9957.813162] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9962.565897] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [ 9970.954510] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9976.020913] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9985.606817] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9985.849494] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9985.863430] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9985.869950] LustreError: 333681:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 9985.877487] LustreError: 333681:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9985.881697] LustreError: 333681:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9985.937192] Lustre: server umount lustre-OST0000 complete [ 9985.941387] LustreError: 333681:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9995.956629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9996.301215] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9996.531430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10003.347391] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [10004.808166] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10010.599480] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10018.192365] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10018.289222] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10018.370418] Lustre: lustre-OST0001: new disk, initializing [10018.375991] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10018.383200] Lustre: 335219:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10018.494384] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10025.291902] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [10026.620688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10026.632024] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10026.749629] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10033.103831] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10037.223082] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10037.225269] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10037.235244] LustreError: Skipped 1 previous similar message [10037.237753] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10037.266103] Lustre: Skipped 1 previous similar message [10042.343891] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10042.360782] Lustre: Skipped 2 previous similar messages [10042.582974] Lustre: server umount lustre-OST0001 complete [10047.460743] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10047.465205] LustreError: 334147:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10047.467773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10047.484504] Lustre: Skipped 1 previous similar message [10047.509108] LustreError: 334147:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10052.589646] LustreError: 334147:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10052.606971] LustreError: 334147:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10053.030416] Lustre: server umount lustre-OST0000 complete [10060.267534] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10060.279697] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10060.299726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10060.310121] Lustre: Skipped 3 previous similar messages [10064.441039] Lustre: server umount lustre-MDT0000 complete [10067.939543] LustreError: 331798:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10067.996343] LustreError: 331798:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10068.923328] LustreError: 331785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787125059 with bad export cookie 13555340349252155995 [10068.930822] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10068.940131] LustreError: 331785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10069.383441] Lustre: server umount lustre-MDT0001 complete [10084.173090] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [10087.009262] Key type lgssc unregistered [10087.240135] LNet: 337179:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10087.273218] LNetError: 337179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10088.296209] LNet: Removed LNI 192.168.201.151@tcp [10089.075416] Key type .llcrypt unregistered [10089.077178] Key type ._llcrypt unregistered [10160.663470] Key type ._llcrypt registered [10160.665366] Key type .llcrypt registered [10160.752901] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [10175.743856] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [10177.751811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10177.835988] alg: No test for adler32 (adler32-zlib) [10179.218252] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [10179.670704] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [10181.320170] Key type lgssc registered [10183.119674] Lustre: Echo OBD driver; http://www.lustre.org/ [10216.841114] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10216.874381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10218.239867] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10218.280167] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10218.401787] Lustre: lustre-MDT0000: new disk, initializing [10218.457623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10218.471422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10222.993563] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [10235.224849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10235.413444] Lustre: 342117:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10235.528576] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10235.535606] Lustre: Skipped 1 previous similar message [10235.694296] Lustre: lustre-MDT0001: new disk, initializing [10235.885246] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10235.957816] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10235.990673] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10242.268119] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [10253.417716] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10258.258820] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10266.675788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10267.055104] Lustre: lustre-OST0000: new disk, initializing [10267.068331] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10267.078675] Lustre: 344087:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10267.202455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10276.255211] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [10276.899775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10276.909299] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10277.033661] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10286.460169] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10291.212249] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10291.434769] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10296.334582] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10296.543807] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10302.439040] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10302.456244] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10302.479149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10302.963274] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10302.971227] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10303.002512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10304.897411] Lustre: server umount lustre-OST0000 complete [10312.680941] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10312.688109] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10312.710485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10314.364743] Lustre: server umount lustre-MDT0000 complete [10318.578658] LustreError: 342110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787125309 with bad export cookie 7422608937956498628 [10318.591905] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10319.051384] Lustre: server umount lustre-MDT0001 complete [10328.276679] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 03:41:58 (1787125318) [10336.083814] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 03:42:06 (1787125326) [10344.672086] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10345.199441] LustreError: 346841:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10345.217173] LustreError: 346841:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10345.282552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10350.242309] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [10350.568727] LustreError: 346842:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10355.683620] LustreError: 346841:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10358.422862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10358.860757] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10362.947721] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [10370.817314] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10376.721839] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10383.800675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10384.163341] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10390.273389] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [10396.568403] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10408.557861] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10410.402454] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10412.311317] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10414.074842] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10416.438399] Lustre: Failing over lustre-MDT0000 [10416.717833] Lustre: server umount lustre-MDT0000 complete [10419.175127] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10419.179431] LustreError: 346838:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10419.191145] Lustre: Skipped 3 previous similar messages [10419.209877] LustreError: 346838:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10424.290736] LustreError: 347569:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10424.327520] LustreError: 347569:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [10434.537065] LustreError: 346841:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10434.570978] LustreError: 346841:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [10435.552461] Lustre: 340341:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787125410/real 1787125410] req@0000000033ab8156 x1873936562330752/t0(0) o400->MGC192.168.201.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787125426 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10435.613676] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10436.763983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10445.800107] LustreError: 340338:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000038b58eb8 x1873936562336640/t0(0) o250->MGC192.168.201.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10446.237842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10446.330930] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10451.244409] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10451.400302] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [10451.448375] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10451.501589] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10451.579350] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10463.265808] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10465.910443] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10470.895942] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10473.936571] Lustre: Failing over lustre-MDT0000 [10476.258783] Lustre: server umount lustre-MDT0000 complete [10476.868656] LustreError: 346836:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10476.894175] LustreError: 346836:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [10477.029043] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10477.049391] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10477.074363] Lustre: Skipped 2 previous similar messages [10492.385052] Lustre: 340339:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787125467/real 1787125468] req@0000000005f475c7 x1873936562354176/t0(0) o400->MGC192.168.201.151@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787125483 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10492.407865] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10497.267663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10502.639029] LustreError: 340338:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000dcad3051 x1873936562359552/t0(0) o250->MGC192.168.201.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10502.966531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10503.038783] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10507.586533] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10508.077747] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [10508.272413] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10508.283789] Lustre: Skipped 2 previous similar messages [10508.340563] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10508.462920] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10517.668965] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10519.490289] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10528.746454] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10528.762191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10528.762976] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10528.766797] Lustre: Skipped 2 previous similar messages [10528.803704] Lustre: Skipped 4 previous similar messages [10533.586970] Lustre: server umount lustre-MDT0000 complete [10533.859586] LustreError: 346841:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10533.875759] LustreError: 346841:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 26 previous similar messages [10537.175774] LustreError: 346822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787125528 with bad export cookie 7422608937956501344 [10537.185805] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10537.189995] LustreError: 346822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10537.584444] Lustre: server umount lustre-MDT0001 complete [10552.742683] Lustre: server umount lustre-OST0000 complete [10568.581346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10569.304617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10573.704149] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [10584.579412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10592.126471] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [10600.683825] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10605.842117] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10614.102619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10614.263699] LustreError: 355025:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10614.291863] LustreError: 355025:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [10619.892636] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10622.882188] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [10633.038405] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10640.376291] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10640.382608] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10640.388423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10640.399077] Lustre: Skipped 1 previous similar message [10640.426124] LustreError: Skipped 1 previous similar message [10645.100304] Lustre: server umount lustre-OST0000 complete [10650.596927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10650.607685] Lustre: Skipped 2 previous similar messages [10655.503186] Lustre: server umount lustre-MDT0000 complete [10658.873870] LustreError: 353187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787125649 with bad export cookie 7422608937956502086 [10658.890652] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10658.898058] LustreError: 353187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10659.324336] Lustre: server umount lustre-MDT0001 complete [10671.659856] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [10675.232393] Key type lgssc unregistered [10675.607516] LNet: 356740:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10675.616507] LNetError: 356740:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10675.632762] LNet: Removed LNI 192.168.201.151@tcp [10676.802882] Key type .llcrypt unregistered [10676.809858] Key type ._llcrypt unregistered [10703.106814] Key type ._llcrypt registered [10703.110783] Key type .llcrypt registered [10703.236268] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [10704.881139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10704.944339] alg: No test for adler32 (adler32-zlib) [10706.155879] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [10706.370939] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [10708.049730] Key type lgssc registered [10709.180344] Lustre: Echo OBD driver; http://www.lustre.org/ [10717.708490] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10717.748943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10719.713754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10726.139837] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [10729.962747] LustreError: 358255:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10729.974658] LustreError: 358255:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10735.078733] LustreError: 358256:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10735.745713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10736.181371] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10742.446724] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [10752.936874] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10759.844503] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10767.455362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10767.950088] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10774.359888] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [10782.354549] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10789.447071] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10804.708381] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10804.711555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10804.729717] Lustre: Skipped 2 previous similar messages [10804.746896] Lustre: Skipped 2 previous similar messages [10809.830236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10809.841471] Lustre: Skipped 2 previous similar messages [10810.092414] Lustre: server umount lustre-MDT0000 complete [10814.259796] LustreError: 358238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787125805 with bad export cookie 13312836769053715935 [10814.269258] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10814.271884] LustreError: 358238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10815.012889] Lustre: server umount lustre-MDT0001 complete [10829.800213] Lustre: server umount lustre-OST0000 complete [10845.720541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10846.465937] LustreError: 361991:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10846.572287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10851.818158] LustreError: 361992:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10852.089294] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [10856.931347] LustreError: 361991:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10862.004761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10862.657093] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10867.997172] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [10877.976810] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10884.844134] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10894.073821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10894.466783] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10903.466949] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [10913.716649] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10917.075489] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10934.241861] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10934.257128] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10934.278778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10937.832110] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10937.847315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10937.851570] Lustre: Skipped 1 previous similar message [10939.406964] Lustre: server umount lustre-MDT0000 complete [10942.948030] LustreError: 362944:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10942.960742] LustreError: 362944:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [10943.533361] LustreError: 361973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787125934 with bad export cookie 13312836769053717048 [10943.560698] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10943.979270] Lustre: server umount lustre-MDT0001 complete [10958.634536] Lustre: server umount lustre-OST0000 complete [10980.678932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10981.666806] LustreError: 365904:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10981.805925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10987.528914] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [10997.193609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11002.837769] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [11010.723112] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11016.775819] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11024.120805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11024.484475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11024.492281] Lustre: Skipped 1 previous similar message [11026.564544] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11031.165483] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [11038.213133] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11043.362756] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11043.365237] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11043.381746] Lustre: Skipped 1 previous similar message [11043.392133] Lustre: Skipped 2 previous similar messages [11048.434674] LustreError: 367724:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11048.451124] LustreError: 367724:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [11048.687769] Lustre: server umount lustre-OST0000 complete [11053.543237] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11053.550169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11053.558729] Lustre: Skipped 1 previous similar message [11053.569683] Lustre: Skipped 1 previous similar message [11058.672435] Lustre: server umount lustre-MDT0000 complete [11062.367904] LustreError: 365886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787126053 with bad export cookie 13312836769053718070 [11062.379871] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11062.381899] LustreError: 365886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [11062.756620] Lustre: server umount lustre-MDT0001 complete [11074.870125] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [11077.326854] Key type lgssc unregistered [11077.595929] LNet: 369437:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11077.608105] LNetError: 369437:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11077.639360] LNet: Removed LNI 192.168.201.151@tcp [11078.482265] Key type .llcrypt unregistered [11078.484216] Key type ._llcrypt unregistered [11106.644740] Key type ._llcrypt registered [11106.649213] Key type .llcrypt registered [11106.752393] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [11107.534255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11107.576653] alg: No test for adler32 (adler32-zlib) [11108.708565] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [11108.891855] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [11110.544194] Key type lgssc registered [11112.360412] Lustre: Echo OBD driver; http://www.lustre.org/ [11122.445453] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11122.491301] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11122.504171] LustreError: 370881:0:(osd_handler.c:8554:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11122.532649] LustreError: 370881:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11122.544819] LustreError: 370881:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11122.565385] LustreError: 370881:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11122.581484] LustreError: 370881:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [11131.343562] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11131.350580] LustreError: 371279:0:(osd_handler.c:8554:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11131.366656] LustreError: 371279:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11131.372202] LustreError: 371279:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11131.377962] LustreError: 371279:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11131.387347] LustreError: 371279:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [11156.182098] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [11159.542619] Key type lgssc unregistered [11160.004729] LNet: 372283:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11160.019444] LNetError: 372283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11160.049307] LNet: Removed LNI 192.168.201.151@tcp [11161.204307] Key type .llcrypt unregistered [11161.211742] Key type ._llcrypt unregistered [11183.137987] Key type ._llcrypt registered [11183.140426] Key type .llcrypt registered [11183.254700] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [11200.591156] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [11201.534899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11201.635087] alg: No test for adler32 (adler32-zlib) [11202.858627] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [11203.074843] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [11204.688127] Key type lgssc registered [11206.025736] Lustre: Echo OBD driver; http://www.lustre.org/ [11241.939237] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11241.979562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11243.298775] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11243.324994] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11243.393604] Lustre: lustre-MDT0000: new disk, initializing [11243.468676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11243.480696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11248.909287] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [11260.834972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11261.014405] Lustre: 375442:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11261.084596] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11261.091747] Lustre: Skipped 1 previous similar message [11261.174479] Lustre: lustre-MDT0001: new disk, initializing [11261.289414] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11261.328083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11261.348293] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11266.497256] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [11276.046959] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11283.452015] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11291.252397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11291.514273] Lustre: lustre-OST0000: new disk, initializing [11291.520321] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11291.526215] Lustre: 377412:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11291.620282] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11297.931323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11297.946350] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11298.065809] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11298.714899] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [11309.015500] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11315.608587] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11315.823794] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11321.362576] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11321.601978] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11328.998310] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11328.999162] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11329.013617] LustreError: Skipped 1 previous similar message [11329.016836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11329.040128] Lustre: Skipped 1 previous similar message [11329.069203] Lustre: Skipped 1 previous similar message [11330.819967] Lustre: server umount lustre-OST0000 complete [11337.186056] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11337.186823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11337.211023] Lustre: Skipped 1 previous similar message [11341.948000] Lustre: server umount lustre-MDT0000 complete [11346.482150] LustreError: 375433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787126337 with bad export cookie 14606194262437019925 [11346.489748] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11346.498276] LustreError: 375433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11347.267651] Lustre: server umount lustre-MDT0001 complete [11357.437928] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11359.253677] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 03:59:09 (1787126349) [11366.974820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11367.491893] LustreError: 379873:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11367.522891] LustreError: 379873:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11367.602329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11372.045272] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [11373.034714] LustreError: 379874:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11377.126425] LustreError: 379873:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11380.251845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11380.740696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11385.884746] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [11392.806033] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11399.054732] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11406.022922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11406.317384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11413.035488] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [11420.938848] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11428.677261] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11428.874719] Lustre: lustre-OST0001: new disk, initializing [11428.879387] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11428.884821] Lustre: 382711:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11428.983885] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11435.975262] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [11437.639983] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11437.657717] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11437.750360] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11446.007069] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11452.741620] Lustre: lustre-OST0001: Not available for connect from 192.168.201.51@tcp (stopping) [11452.759113] Lustre: Skipped 1 previous similar message [11453.421473] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11453.422055] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11453.455271] LustreError: Skipped 1 previous similar message [11455.711122] Lustre: server umount lustre-OST0001 complete [11457.852503] LustreError: 383147:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11463.663727] LustreError: 383147:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11463.688722] LustreError: 383147:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11473.890774] LustreError: 381694:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11473.896639] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11473.913688] LustreError: 381694:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11473.928280] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11473.942395] Lustre: Skipped 1 previous similar message [11473.947336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11473.957393] Lustre: Skipped 3 previous similar messages [11479.010143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11479.026381] Lustre: Skipped 1 previous similar message [11480.067265] Lustre: server umount lustre-OST0000 complete [11488.242101] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11488.258109] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11488.282123] Lustre: Skipped 1 previous similar message [11488.309970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11490.607694] Lustre: server umount lustre-MDT0000 complete [11494.372161] LustreError: 379868:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11494.399044] LustreError: 379868:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11494.528760] LustreError: 383957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787126485 with bad export cookie 14606194262437020807 [11494.541315] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11494.542781] LustreError: 383957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11501.110475] Lustre: server umount lustre-MDT0001 complete [11513.282242] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [11516.826463] Key type lgssc unregistered [11517.268378] LNet: 384709:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11517.279030] LNetError: 384709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11517.315474] LNet: Removed LNI 192.168.201.151@tcp [11518.398228] Key type .llcrypt unregistered [11518.399990] Key type ._llcrypt unregistered [11572.605408] Key type ._llcrypt registered [11572.606926] Key type .llcrypt registered [11572.730788] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [11573.485118] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11573.571758] alg: No test for adler32 (adler32-zlib) [11574.834655] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [11575.047859] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [11576.688237] Key type lgssc registered [11577.978505] Lustre: Echo OBD driver; http://www.lustre.org/ [11585.940088] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11585.989056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11587.256970] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11587.271833] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11587.390030] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11592.140537] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [11601.192399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11601.277841] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11601.294399] Lustre: 387509:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11601.577778] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11607.049763] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [11614.841355] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11620.886279] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11626.977665] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11626.985451] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11627.007973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11627.497061] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11627.532779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11627.540429] Lustre: Skipped 1 previous similar message [11629.557341] Lustre: server umount lustre-MDT0000 complete [11633.402441] LustreError: 387501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787126624 with bad export cookie 13423183028429550807 [11633.415029] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11633.431661] LustreError: 387501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11633.851214] Lustre: server umount lustre-MDT0001 complete [11644.618667] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 04:03:54 (1787126634) [11651.358893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11651.740073] LustreError: 390021:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11651.750707] LustreError: 390021:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11651.782884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11656.410982] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [11657.186137] LustreError: 390022:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11662.305332] LustreError: 390021:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11664.584641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11665.144109] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11670.317483] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [11677.764944] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11683.406503] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11690.630766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11690.872919] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11691.157876] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11697.984323] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [11705.653858] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11713.505073] Lustre: Permanently deactivating lustre-OST0000 [11713.516344] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11723.296730] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11725.733603] LustreError: 393057:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11725.741314] LustreError: 393057:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [11725.745811] LustreError: 393057:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11729.715035] LustreError: 393106:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [11729.720450] LustreError: 393106:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11734.260574] Lustre: server umount lustre-OST0000 complete [11738.619523] LustreError: 393355:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [11738.631108] LustreError: 393355:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11742.177181] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11742.190854] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11742.214033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11748.833968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11748.857318] Lustre: Skipped 3 previous similar messages [11751.257077] Lustre: server umount lustre-MDT0000 complete [11753.963404] LustreError: 392523:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11753.970834] LustreError: 392523:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11755.139523] LustreError: 392714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787126746 with bad export cookie 13423183028429551388 [11755.155536] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11765.877834] Lustre: server umount lustre-MDT0001 complete [11775.621138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11779.890647] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [11783.338743] LustreError: 394517:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11789.631741] Lustre: 394856:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.151@tcp [11791.882605] Lustre: server umount MGS complete [11804.070765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11804.549669] LustreError: 395580:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11804.584584] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11804.591745] Lustre: Skipped 1 previous similar message [11804.704977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11810.746770] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [11814.882417] LustreError: 395580:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11814.901386] LustreError: 395580:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11820.761666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11821.122177] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11821.216588] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11826.500973] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [11835.353757] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11841.179911] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11843.026831] Lustre: Permanently reactivating lustre-OST0000 [11843.049193] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11843.055951] Lustre: Skipped 2 previous similar messages [11870.180312] LustreError: 397323:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11879.366673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11879.765642] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11886.743800] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [11895.094702] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11901.410805] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11901.412501] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11901.430561] Lustre: Skipped 1 previous similar message [11901.434499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11901.435256] LustreError: Skipped 1 previous similar message [11901.463049] Lustre: Skipped 1 previous similar message [11903.536291] Lustre: server umount lustre-OST0000 complete [11908.082726] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11908.098421] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11908.126244] Lustre: Skipped 1 previous similar message [11911.655791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11911.664804] Lustre: Skipped 2 previous similar messages [11913.669286] Lustre: server umount lustre-MDT0000 complete [11916.768935] LustreError: 395580:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11916.781436] LustreError: 395580:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11918.593522] LustreError: 395560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787126909 with bad export cookie 13423183028429552438 [11918.598895] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11918.610256] LustreError: 395560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [11919.066652] Lustre: server umount lustre-MDT0001 complete [11930.083959] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [11932.609698] Key type lgssc unregistered [11932.865078] LNet: 399947:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11932.873439] LNetError: 399947:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11932.893740] LNet: Removed LNI 192.168.201.151@tcp [11933.933120] Key type .llcrypt unregistered [11933.934930] Key type ._llcrypt unregistered [11951.546418] Key type ._llcrypt registered [11951.556194] Key type .llcrypt registered [11951.735627] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [11966.956650] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [11968.444956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11968.464974] alg: No test for adler32 (adler32-zlib) [11969.548695] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [11969.810649] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [11971.458326] Key type lgssc registered [11972.789671] Lustre: Echo OBD driver; http://www.lustre.org/ [12010.481325] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12010.518707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12011.850582] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12011.930559] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12012.142255] Lustre: lustre-MDT0000: new disk, initializing [12012.314924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12012.407466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12019.398372] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [12031.244673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12031.404163] Lustre: 403102:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12031.477744] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12031.482299] Lustre: Skipped 1 previous similar message [12031.588489] Lustre: lustre-MDT0001: new disk, initializing [12031.728349] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12031.765608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12031.790702] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12036.751929] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [12046.921674] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12052.413996] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12059.204712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12059.588363] Lustre: lustre-OST0000: new disk, initializing [12059.593907] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12059.598785] Lustre: 405074:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12059.713164] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12065.437496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12065.466135] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12065.664723] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12067.465389] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [12076.527255] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12081.006670] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12081.263958] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12085.368790] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12085.578203] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12089.328542] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12089.354367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12089.356569] Lustre: Skipped 1 previous similar message [12089.380300] Lustre: Skipped 1 previous similar message [12093.435959] Lustre: server umount lustre-OST0000 complete [12098.529798] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12098.545631] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12098.560920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12099.564608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12099.566151] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12103.710981] Lustre: server umount lustre-MDT0000 complete [12107.517871] LustreError: 403094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787127098 with bad export cookie 15673246700922832585 [12107.522024] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12107.525186] LustreError: 403094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12107.921373] Lustre: server umount lustre-MDT0001 complete [12116.469479] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12118.194552] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 04:11:48 (1787127108) [12125.282075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12125.752478] LustreError: 407537:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12125.774166] LustreError: 407537:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12125.818546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12130.916706] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [12131.299482] LustreError: 407538:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12135.402720] LustreError: 407537:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12140.514752] LustreError: 407538:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12140.621416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12141.307450] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12146.490851] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [12155.001282] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12160.893778] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12168.117841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12168.477859] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12175.453284] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [12184.342103] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12186.476981] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12189.556895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12196.322796] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12196.335272] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12196.353981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12196.356493] Lustre: Skipped 1 previous similar message [12197.368970] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12202.219932] Lustre: server umount lustre-OST0000 complete [12207.589714] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12207.606074] Lustre: Skipped 1 previous similar message [12207.612489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12207.615292] Lustre: Skipped 3 previous similar messages [12213.475594] Lustre: server umount lustre-MDT0000 complete [12217.376404] LustreError: 407518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787127208 with bad export cookie 15673246700922833467 [12217.377269] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12217.391628] LustreError: 407518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12218.040860] Lustre: server umount lustre-MDT0001 complete [12230.580850] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [12233.708608] Key type lgssc unregistered [12234.105523] LNet: 411263:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12234.131945] LNetError: 411263:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12235.177272] LNet: Removed LNI 192.168.201.151@tcp [12236.190622] Key type .llcrypt unregistered [12236.206941] Key type ._llcrypt unregistered [12264.196982] Key type ._llcrypt registered [12264.199192] Key type .llcrypt registered [12264.313244] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [12264.595298] Key type .llcrypt unregistered [12264.598844] Key type ._llcrypt unregistered [12283.087800] Key type ._llcrypt registered [12283.090479] Key type .llcrypt registered [12283.202582] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [12284.270515] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12284.463056] alg: No test for adler32 (adler32-zlib) [12285.594405] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [12285.813579] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [12287.480161] Key type lgssc registered [12288.661466] Lustre: Echo OBD driver; http://www.lustre.org/ [12296.710206] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12296.752907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12298.292730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12302.620840] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [12308.450371] LustreError: 413477:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12308.467111] LustreError: 413477:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12311.805577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12312.126181] LustreError: 413476:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12312.149116] LustreError: 413476:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12312.155155] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12317.156924] LustreError: 413473:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12317.175324] LustreError: 413473:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12317.857559] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [12322.274854] LustreError: 414219:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12322.287505] LustreError: 414219:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12323.311981] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12324.840055] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12328.437471] LustreError: 413473:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12328.461710] LustreError: 413473:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12332.407504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12332.685547] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12336.254850] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [12346.004544] Lustre: server umount lustre-OST0000 complete [12351.978096] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12351.991175] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12352.003503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12353.507371] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12353.508989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12353.524988] Lustre: Skipped 1 previous similar message [12358.632689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12358.639029] Lustre: Skipped 1 previous similar message [12359.757570] Lustre: server umount lustre-MDT0000 complete [12363.200926] LustreError: 413458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787127354 with bad export cookie 7155898696305973524 [12363.203961] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12363.218146] LustreError: 413458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12373.666287] Lustre: server umount lustre-MDT0001 complete [12385.283373] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [12387.798247] Key type lgssc unregistered [12388.104708] LNet: 416625:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12388.108553] LNetError: 416625:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12388.117975] LNet: Removed LNI 192.168.201.151@tcp [12389.020955] Key type .llcrypt unregistered [12389.023132] Key type ._llcrypt unregistered [12416.186259] Key type ._llcrypt registered [12416.188314] Key type .llcrypt registered [12416.280693] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [12417.765506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12417.832803] alg: No test for adler32 (adler32-zlib) [12418.960759] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [12419.294787] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [12421.041315] Key type lgssc registered [12421.973416] Lustre: Echo OBD driver; http://www.lustre.org/ [12429.983351] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12430.008174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12445.728551] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12450.850609] LustreError: 418018:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.151@tcp: failed processing log, type 4: rc = -110 [12477.536368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12484.636185] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [12493.261383] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12500.832789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12501.389782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12501.468412] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12505.811231] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [12506.597663] LustreError: 418044:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12506.623540] LustreError: 418044:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12511.720612] LustreError: 418043:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12512.741520] LustreError: 418044:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12512.762254] LustreError: 418044:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12514.824356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12515.282361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12520.138597] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [12529.411416] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12536.785551] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12546.033508] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12546.052516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12546.052731] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12546.061953] Lustre: Skipped 1 previous similar message [12546.083712] Lustre: Skipped 1 previous similar message [12550.614204] Lustre: server umount lustre-OST0000 complete [12556.258852] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12556.261759] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12556.273609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12556.291340] Lustre: Skipped 1 previous similar message [12560.603815] Lustre: server umount lustre-MDT0000 complete [12561.382846] LustreError: 419057:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12561.421923] LustreError: 419057:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12564.935416] LustreError: 420546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787127555 with bad export cookie 14996429698198602708 [12564.938158] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12564.955931] LustreError: 420546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12565.542930] Lustre: server umount lustre-MDT0001 complete [12577.356661] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [12580.056333] Key type lgssc unregistered [12580.420222] LNet: 421587:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12580.427796] LNetError: 421587:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12580.460242] LNet: Removed LNI 192.168.201.151@tcp [12581.507372] Key type .llcrypt unregistered [12581.509960] Key type ._llcrypt unregistered [12608.050243] Key type ._llcrypt registered [12608.053626] Key type .llcrypt registered [12608.199141] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [12609.984909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12610.337491] alg: No test for adler32 (adler32-zlib) [12611.779789] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [12612.238604] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [12614.129908] Key type lgssc registered [12616.719303] Lustre: Echo OBD driver; http://www.lustre.org/ [12630.042763] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12630.062362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12631.656939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12637.845224] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [12641.262580] LustreError: 423019:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12641.280612] LustreError: 423019:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12646.376254] LustreError: 423020:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12648.173281] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12648.483566] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12653.606730] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [12661.765847] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12668.017617] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12675.224068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12675.623054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12676.674449] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12681.946796] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [12689.692083] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12694.497954] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12694.508492] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12694.544985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12697.432188] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.51@tcp (stopping) [12699.106791] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12699.110508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12699.129495] Lustre: Skipped 1 previous similar message [12699.138037] Lustre: Skipped 2 previous similar messages [12700.896394] Lustre: server umount lustre-MDT0000 complete [12702.507060] LustreError: 424859:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.51@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12703.250242] LustreError: 423001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.51@tcp arrived at 1787127694 with bad export cookie 15672937469894728670 [12703.266265] LustreError: 423001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12705.496350] LustreError: 422999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787127696 with bad export cookie 15672937469894727781 [12705.504172] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12705.512804] LustreError: 422999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12709.347881] LustreError: 423016:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12709.350182] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12709.370621] LustreError: 423016:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [12709.404160] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12709.438549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12711.968519] Lustre: server umount lustre-OST0000 complete [12715.505462] LustreError: 423020:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12715.518700] LustreError: 423020:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [12719.812819] Lustre: server umount lustre-MDT0001 complete [12732.636279] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [12736.088156] Key type lgssc unregistered [12736.370807] LNet: 426708:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12736.378386] LNetError: 426708:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12737.445911] LNet: Removed LNI 192.168.201.151@tcp [12738.538993] Key type .llcrypt unregistered [12738.544876] Key type ._llcrypt unregistered [12765.239276] Key type ._llcrypt registered [12765.244965] Key type .llcrypt registered [12765.385578] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [12766.443489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12766.599183] alg: No test for adler32 (adler32-zlib) [12767.797635] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [12768.162733] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [12769.928213] Key type lgssc registered [12771.274185] Lustre: Echo OBD driver; http://www.lustre.org/ [12778.754881] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12778.799851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12780.263599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12784.422774] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [12790.766342] LustreError: 428127:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12790.775566] LustreError: 428127:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12793.784686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12794.300599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12798.816401] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [12806.491824] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12812.351812] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12821.965709] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12822.340948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12825.389427] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12829.394942] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [12839.862273] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12845.030846] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12845.051889] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12845.055536] Lustre: Skipped 1 previous similar message [12846.948822] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.51@tcp (stopping) [12849.891575] Lustre: server umount lustre-MDT0001 complete [12850.164988] LustreError: 428123:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12850.180587] LustreError: 428123:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12855.270418] LustreError: 428128:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12855.287427] LustreError: 428128:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12856.289375] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12856.295344] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12856.312210] Lustre: Skipped 1 previous similar message [12856.317763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12860.390087] LustreError: 428866:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12860.393210] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12862.261647] Lustre: server umount lustre-OST0000 complete [12865.509465] LustreError: 428128:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12867.073274] Lustre: server umount lustre-MDT0000 complete [12883.873967] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [12886.714208] Key type lgssc unregistered [12886.955945] LNet: 431812:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12886.963859] LNetError: 431812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12887.977352] LNet: Removed LNI 192.168.201.151@tcp [12889.034214] Key type .llcrypt unregistered [12889.038986] Key type ._llcrypt unregistered [12915.008362] Key type ._llcrypt registered [12915.010959] Key type .llcrypt registered [12915.164659] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [12930.799498] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [12932.030647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12932.228278] alg: No test for adler32 (adler32-zlib) [12933.268307] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [12933.508216] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [12935.201604] Key type lgssc registered [12936.939775] Lustre: Echo OBD driver; http://www.lustre.org/ [12976.666290] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [12993.482032] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12993.514540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12994.767236] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12994.791680] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12994.885318] Lustre: lustre-MDT0000: new disk, initializing [12994.989083] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12995.005216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12999.964077] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [13016.991058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13017.325737] Lustre: lustre-OST0000: new disk, initializing [13017.329289] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13017.335104] Lustre: Skipped 1 previous similar message [13017.347829] Lustre: 436852:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13017.458976] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13024.430698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13024.451522] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13024.493269] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13025.319439] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [13036.691491] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13053.727812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13053.866160] Lustre: 435834:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13053.932630] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13053.988436] Lustre: lustre-MDT0001: new disk, initializing [13054.093715] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13054.138419] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13054.151771] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13058.966759] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [13071.316632] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13071.613167] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13077.370249] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13077.621423] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13085.502301] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.51@tcp (stopping) [13086.184529] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13086.187555] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13086.211439] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13089.398894] Lustre: server umount lustre-MDT0001 complete [13096.421447] LustreError: 435843:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13096.451832] LustreError: 435843:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [13101.538505] LustreError: 435840:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13101.567290] LustreError: 435840:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13106.675181] LustreError: 435838:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13111.778164] LustreError: 439337:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13111.779055] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13111.797874] LustreError: 439337:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13111.814570] Lustre: Skipped 1 previous similar message [13111.824306] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13111.824934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13111.833271] Lustre: Skipped 1 previous similar message [13113.988062] Lustre: server umount lustre-OST0000 complete [13116.901224] LustreError: 435844:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13124.547377] Lustre: server umount lustre-MDT0000 complete [13137.817610] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [13140.847201] Key type lgssc unregistered [13141.218166] LNet: 440600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13141.226942] LNetError: 440600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13141.240535] LNet: Removed LNI 192.168.201.151@tcp [13142.140785] Key type .llcrypt unregistered [13142.146708] Key type ._llcrypt unregistered [13191.709653] Key type ._llcrypt registered [13191.719335] Key type .llcrypt registered [13191.881152] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [13192.407263] Key type .llcrypt unregistered [13192.411290] Key type ._llcrypt unregistered [13212.257551] Key type ._llcrypt registered [13212.259509] Key type .llcrypt registered [13212.368028] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [13213.783257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13213.817076] alg: No test for adler32 (adler32-zlib) [13215.015315] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [13215.270742] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [13217.008196] Key type lgssc registered [13217.964823] Lustre: Echo OBD driver; http://www.lustre.org/ [13247.077243] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13247.162578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13248.484856] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13248.547816] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13248.641271] Lustre: lustre-MDT0000: new disk, initializing [13248.717402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13248.735517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13254.014263] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [13265.366197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13265.517323] Lustre: 443982:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13265.577753] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13265.582547] Lustre: Skipped 1 previous similar message [13265.688781] Lustre: lustre-MDT0001: new disk, initializing [13265.820346] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13265.866701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13265.878492] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13270.838141] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [13282.901100] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13290.767415] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13297.618235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13297.938625] Lustre: lustre-OST0000: new disk, initializing [13297.942705] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13297.948378] Lustre: 445953:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13298.084295] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13304.697268] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [13305.450809] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13305.466974] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13305.576386] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13315.033598] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13322.209662] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13322.215305] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13322.244665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13326.318902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13326.320187] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13326.327105] Lustre: Skipped 1 previous similar message [13326.343546] Lustre: Skipped 1 previous similar message [13328.138378] Lustre: server umount lustre-MDT0000 complete [13331.838343] LustreError: 443975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787128322 with bad export cookie 2300266801589661599 [13331.855124] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13332.478628] Lustre: server umount lustre-MDT0001 complete [13347.178742] Lustre: server umount lustre-OST0000 complete [13358.170225] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 04:32:28 (1787128348) [13367.225509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13368.034233] LustreError: 447981:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13368.068076] LustreError: 447981:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13368.123291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13373.415765] LustreError: 447982:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13373.723198] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [13378.538484] LustreError: 447981:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13383.359283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13383.611217] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13388.825373] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [13396.051532] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13403.508711] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13424.690944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13424.969133] Lustre: Found index 0 for lustre-OST0000, updating log [13425.176035] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13433.448497] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [13439.589738] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13442.079154] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13448.674506] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13448.684399] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13448.708081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13448.720677] Lustre: Skipped 1 previous similar message [13453.944893] Lustre: server umount lustre-OST0000 complete [13471.717040] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13471.729194] Lustre: Skipped 1 previous similar message [13471.741071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13471.755259] Lustre: Skipped 3 previous similar messages [13476.835741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13476.854514] Lustre: Skipped 1 previous similar message [13477.735740] Lustre: server umount lustre-MDT0000 complete [13477.876274] LustreError: 447981:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13481.492897] LustreError: 447961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787128472 with bad export cookie 2300266801589662999 [13481.495120] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13481.499404] LustreError: 447961:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [13481.839480] Lustre: server umount lustre-MDT0001 complete [13491.667078] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 04:34:41 (1787128481) [13491.809502] LNet: 1 local NIs in recovery (showing 1): 192.168.201.151@tcp [14113.904807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14114.504179] LustreError: 454399:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14114.537483] LustreError: 454399:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [14114.629928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14119.928251] LustreError: 454400:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14120.240245] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [14125.027329] LustreError: 454399:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14129.260226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14129.865170] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14135.547330] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [14143.823489] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14152.313381] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14169.760163] LNet: 1 local NIs in recovery (showing 1): 192.168.201.151@tcp [14174.004551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14174.220139] Lustre: Found index 0 for lustre-OST0000, updating log [14174.524820] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14181.104072] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [14190.384724] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14193.122367] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14193.151660] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14193.167893] Lustre: Skipped 1 previous similar message [14193.185637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14199.374061] Lustre: server umount lustre-OST0000 complete [14204.402820] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14204.419373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14206.510685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14206.516277] Lustre: Skipped 4 previous similar messages [14209.571304] LustreError: 454400:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14209.598786] LustreError: 454400:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14209.662848] Lustre: server umount lustre-MDT0000 complete [14213.128117] LustreError: 454380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787129204 with bad export cookie 2300266801589664084 [14213.137747] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14213.139940] LustreError: 454380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14213.397311] Lustre: server umount lustre-MDT0001 complete [14229.156887] LNet: 1 local NIs in recovery (showing 1): 192.168.201.151@tcp [14246.115962] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [14256.171466] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [14291.895297] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 04:48:01 (1787129281) [14312.009793] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [14315.207848] Key type lgssc unregistered [14315.706610] LNet: 461012:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14315.710576] LNetError: 461012:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14435.574631] LNet: Removed LNI 192.168.201.151@tcp [14436.673695] Key type .llcrypt unregistered [14436.677829] Key type ._llcrypt unregistered [14454.164959] Key type ._llcrypt registered [14454.167680] Key type .llcrypt registered [14454.298165] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [14455.306321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14455.676957] alg: No test for adler32 (adler32-zlib) [14456.822656] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [14457.121478] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [14458.792126] Key type lgssc registered [14460.273186] Lustre: Echo OBD driver; http://www.lustre.org/ [14466.169934] LNet: Added LNI 192.168.201.151@tcp100 [8/256/0/180] [14467.679665] LNet: Added LNI 192.168.201.151@tcp101 [8/256/0/180] [14468.745731] LNet: Added LNI 192.168.201.151@tcp102 [8/256/0/180] [14469.738223] LNet: Added LNI 192.168.201.151@tcp103 [8/256/0/180] [14470.612634] LNet: Added LNI 192.168.201.151@tcp104 [8/256/0/180] [14471.447354] LNet: Added LNI 192.168.201.151@tcp105 [8/256/0/180] [14472.506861] LNet: Added LNI 192.168.201.151@tcp106 [8/256/0/180] [14473.473356] LNet: Added LNI 192.168.201.151@tcp107 [8/256/0/180] [14474.305954] LNet: Added LNI 192.168.201.151@tcp108 [8/256/0/180] [14475.274206] LNet: Added LNI 192.168.201.151@tcp109 [8/256/0/180] [14476.290388] LNet: Added LNI 192.168.201.151@tcp110 [8/256/0/180] [14477.221962] LNet: Added LNI 192.168.201.151@tcp111 [8/256/0/180] [14478.185348] LNet: Added LNI 192.168.201.151@tcp112 [8/256/0/180] [14479.418934] LNet: Added LNI 192.168.201.151@tcp113 [8/256/0/180] [14480.920566] LNet: Added LNI 192.168.201.151@tcp114 [8/256/0/180] [14482.095839] LNet: Added LNI 192.168.201.151@tcp115 [8/256/0/180] [14483.552387] LNet: Added LNI 192.168.201.151@tcp116 [8/256/0/180] [14485.031510] LNet: Added LNI 192.168.201.151@tcp117 [8/256/0/180] [14486.156383] LNet: Added LNI 192.168.201.151@tcp118 [8/256/0/180] [14487.663865] LNet: Added LNI 192.168.201.151@tcp119 [8/256/0/180] [14488.905838] LNet: Added LNI 192.168.201.151@tcp120 [8/256/0/180] [14489.724390] LNet: Added LNI 192.168.201.151@tcp121 [8/256/0/180] [14490.455950] LNet: Added LNI 192.168.201.151@tcp122 [8/256/0/180] [14491.404879] LNet: Added LNI 192.168.201.151@tcp123 [8/256/0/180] [14492.516415] LNet: Added LNI 192.168.201.151@tcp124 [8/256/0/180] [14493.660801] LNet: Added LNI 192.168.201.151@tcp125 [8/256/0/180] [14494.574432] LNet: Added LNI 192.168.201.151@tcp126 [8/256/0/180] [14495.601705] LNet: Added LNI 192.168.201.151@tcp127 [8/256/0/180] [14496.596623] LNet: Added LNI 192.168.201.151@tcp128 [8/256/0/180] [14497.374099] LNet: Added LNI 192.168.201.151@tcp129 [8/256/0/180] [14498.243271] LNet: Added LNI 192.168.201.151@tcp130 [8/256/0/180] [14499.046968] LNet: Added LNI 192.168.201.151@tcp131 [8/256/0/180] [14499.975994] LNet: Added LNI 192.168.201.151@tcp132 [8/256/0/180] [14501.070255] LNet: Added LNI 192.168.201.151@tcp133 [8/256/0/180] [14501.934472] LNet: Added LNI 192.168.201.151@tcp134 [8/256/0/180] [14502.860773] LNet: Added LNI 192.168.201.151@tcp135 [8/256/0/180] [14511.341997] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14511.442715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14512.602502] Lustre: 464047:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.151@tcp: can't add NID 192.168.201.151@tcp130: rc = -75 [14512.713919] LNetError: 461505:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.151 [14512.737521] LNetError: Protocol error connecting to 192.168.201.151@tcp120 at host 192.168.201.151:988: is it running a compatible version of Lustre? [14512.762263] LNetError: Skipped 9 previous similar messages [14512.870608] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14512.904674] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14513.009223] Lustre: lustre-MDT0000: new disk, initializing [14513.104938] Lustre: 464103:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.151@tcp: can't add NID 192.168.201.151@tcp130: rc = -75 [14513.116229] Lustre: 464103:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14513.140346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14513.173790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14513.450573] LNet: 10 local NIs in recovery (showing 10): 192.168.201.151@tcp121, 192.168.201.151@tcp122, 192.168.201.151@tcp124, 192.168.201.151@tcp126, 192.168.201.151@tcp125, 192.168.201.151@tcp123, 192.168.201.151@tcp127, 192.168.201.151@tcp128, 192.168.201.151@tcp129, 192.168.201.151@tcp120 [14513.456517] LNetError: 461507:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.151 [14513.507604] LNetError: 461507:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 11 previous similar messages [14513.536112] LNetError: Protocol error connecting to 192.168.201.151@tcp104 at host 192.168.201.151:988: is it running a compatible version of Lustre? [14514.529960] LNetError: 464068:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.151 [14514.554967] LNetError: Protocol error connecting to 192.168.201.151@tcp123 at host 192.168.201.151:988: is it running a compatible version of Lustre? [14518.728620] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [14531.424537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14531.738983] Lustre: 464062:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14531.848666] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14531.855850] Lustre: Skipped 1 previous similar message [14531.992442] Lustre: lustre-MDT0001: new disk, initializing [14532.061261] Lustre: 464866:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.151@tcp: can't add NID 192.168.201.151@tcp130: rc = -75 [14532.076857] Lustre: 464866:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14532.319770] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14532.371082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14532.388352] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14538.293551] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [14540.974438] Lustre: 465218:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.151@tcp: can't add NID 192.168.201.151@tcp130: rc = -75 [14540.992761] Lustre: 465218:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14544.165202] LNetError: 461504:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.151 [14544.190114] LNetError: Protocol error connecting to 192.168.201.151@tcp104 at host 192.168.201.151:988: is it running a compatible version of Lustre? [14548.258959] LNetError: 464067:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.151 [14548.273464] LNetError: 464067:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 1 previous similar message [14548.287217] LNetError: Protocol error connecting to 192.168.201.151@tcp118 at host 192.168.201.151:988: is it running a compatible version of Lustre? [14548.299339] LNetError: Skipped 1 previous similar message [14550.149891] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14556.980772] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14557.479208] LNetError: 464076:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.151 [14557.493194] LNetError: 464076:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 1 previous similar message [14557.499839] LNetError: Protocol error connecting to 192.168.201.151@tcp104 at host 192.168.201.151:988: is it running a compatible version of Lustre? [14557.514241] LNetError: Skipped 1 previous similar message [14564.310993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14564.635614] Lustre: lustre-OST0000: new disk, initializing [14564.640744] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14564.647918] Lustre: 466042:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14564.720227] Lustre: 466047:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.151@tcp: can't add NID 192.168.201.151@tcp130: rc = -75 [14564.728861] Lustre: 466047:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14564.791801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14570.617869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14570.628575] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14570.809666] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14571.357750] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [14573.858775] LNetError: 464070:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.151 [14573.878568] LNetError: 464070:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 4 previous similar messages [14573.890581] LNetError: Protocol error connecting to 192.168.201.151@tcp123 at host 192.168.201.151:988: is it running a compatible version of Lustre? [14573.908044] LNetError: Skipped 4 previous similar messages [14581.944699] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14591.463552] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14591.481633] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14591.485023] LustreError: Skipped 1 previous similar message [14591.486115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14591.527721] Lustre: Skipped 1 previous similar message [14595.475074] Lustre: server umount lustre-OST0000 complete [14601.698630] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14601.719967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14601.733064] Lustre: Skipped 2 previous similar messages [14606.742762] Lustre: server umount lustre-MDT0000 complete [14611.042249] LustreError: 464055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787129602 with bad export cookie 3726207730222172230 [14611.050923] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14611.065287] LustreError: 464055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14611.465677] Lustre: server umount lustre-MDT0001 complete [14612.835582] LNetError: 464074:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.151 [14612.858221] LNetError: 464074:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 6 previous similar messages [14612.874147] LNetError: Protocol error connecting to 192.168.201.151@tcp118 at host 192.168.201.151:988: is it running a compatible version of Lustre? [14612.893971] LNetError: Skipped 6 previous similar messages [14625.418534] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [14629.008222] Key type lgssc unregistered [14629.449640] LNet: 467956:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14629.461729] LNetError: 467956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14629.477154] LNet: Removed LNI 192.168.201.151@tcp [14630.496188] LNet: Removed LNI 192.168.201.151@tcp100 [14630.516531] LNetError: 467956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14630.526493] LNetError: 467956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14630.532768] LNet: Removed LNI 192.168.201.151@tcp101 [14631.584317] LNet: Removed LNI 192.168.201.151@tcp102 [14631.589235] LNetError: 467956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14631.601712] LNetError: 467956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14631.614885] LNet: Removed LNI 192.168.201.151@tcp103 [14632.672168] LNet: Removed LNI 192.168.201.151@tcp104 [14633.696181] LNet: Removed LNI 192.168.201.151@tcp105 [14633.699848] LNetError: 467956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14633.711667] LNetError: 467956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14633.726699] LNet: Removed LNI 192.168.201.151@tcp106 [14634.785257] LNet: Removed LNI 192.168.201.151@tcp107 [14635.809190] LNet: Removed LNI 192.168.201.151@tcp108 [14636.832283] LNet: Removed LNI 192.168.201.151@tcp109 [14637.856368] LNet: Removed LNI 192.168.201.151@tcp110 [14637.861752] LNetError: 467956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14637.867516] LNetError: 467956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14637.879949] LNet: Removed LNI 192.168.201.151@tcp111 [14638.944230] LNet: Removed LNI 192.168.201.151@tcp112 [14639.968483] LNet: Removed LNI 192.168.201.151@tcp113 [14640.992155] LNet: Removed LNI 192.168.201.151@tcp114 [14642.016327] LNet: Removed LNI 192.168.201.151@tcp115 [14643.040183] LNet: Removed LNI 192.168.201.151@tcp116 [14644.064831] LNet: Removed LNI 192.168.201.151@tcp117 [14645.152220] LNet: Removed LNI 192.168.201.151@tcp118 [14646.176263] LNet: Removed LNI 192.168.201.151@tcp119 [14646.179192] LNetError: 467956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14646.187485] LNetError: 467956:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14646.198132] LNet: Removed LNI 192.168.201.151@tcp120 [14647.264223] LNet: Removed LNI 192.168.201.151@tcp121 [14648.288805] LNet: Removed LNI 192.168.201.151@tcp122 [14649.312250] LNet: Removed LNI 192.168.201.151@tcp123 [14650.336237] LNet: Removed LNI 192.168.201.151@tcp124 [14651.360284] LNet: Removed LNI 192.168.201.151@tcp125 [14652.384306] LNet: Removed LNI 192.168.201.151@tcp126 [14653.408293] LNet: Removed LNI 192.168.201.151@tcp127 [14654.433604] LNet: Removed LNI 192.168.201.151@tcp128 [14655.456698] LNet: Removed LNI 192.168.201.151@tcp129 [14655.458343] LNet: Removed LNI 192.168.201.151@tcp130 [14655.463391] LNet: Removed LNI 192.168.201.151@tcp131 [14655.471755] LNet: Removed LNI 192.168.201.151@tcp132 [14655.481931] LNet: Removed LNI 192.168.201.151@tcp133 [14655.490464] LNet: Removed LNI 192.168.201.151@tcp134 [14655.504853] LNet: Removed LNI 192.168.201.151@tcp135 [14656.223611] Key type .llcrypt unregistered [14656.229279] Key type ._llcrypt unregistered [14681.989817] Key type ._llcrypt registered [14681.991698] Key type .llcrypt registered [14682.091462] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [14696.600600] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [14697.608790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14697.712306] alg: No test for adler32 (adler32-zlib) [14698.799230] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [14699.022309] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [14700.656263] Key type lgssc registered [14701.975978] Lustre: Echo OBD driver; http://www.lustre.org/ [14741.577063] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 04:55:31 (1787129731) [14759.976696] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [14763.080837] Key type lgssc unregistered [14763.388500] LNet: 472045:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14763.395150] LNetError: 472045:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14763.412379] LNet: Removed LNI 192.168.201.151@tcp [14764.491348] Key type .llcrypt unregistered [14764.493232] Key type ._llcrypt unregistered [14782.017420] Key type ._llcrypt registered [14782.019168] Key type .llcrypt registered [14782.183388] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [14784.107915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14784.149052] alg: No test for adler32 (adler32-zlib) [14785.457515] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [14785.898768] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [14787.648269] Key type lgssc registered [14789.056559] Lustre: Echo OBD driver; http://www.lustre.org/ [14799.495837] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14799.530595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14801.085621] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14801.164237] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14801.306787] Lustre: lustre-MDT0000: new disk, initializing [14801.457717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14801.485413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14807.012974] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [14818.283099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14818.475992] Lustre: 473257:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14818.598286] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14818.605907] Lustre: Skipped 1 previous similar message [14818.710201] Lustre: lustre-MDT0001: new disk, initializing [14818.869205] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14818.944234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14818.957293] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14824.721440] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [14835.087959] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14840.898612] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14848.638340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14849.013949] Lustre: lustre-OST0000: new disk, initializing [14849.017606] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14849.022988] Lustre: 475227:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14849.125288] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14855.992640] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [14858.915990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14858.929466] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14859.091313] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14865.001124] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14869.186295] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing lnet_if_list [14874.754446] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14874.789404] LNet: Added LNI 192.168.201.151@tcp10 [8/256/0/180] [14880.689468] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14880.720281] LNet: Added LNI 192.168.201.151@tcp11 [8/256/0/180] [14885.394383] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14885.418627] LNet: Added LNI 192.168.201.151@tcp12 [8/256/0/180] [14890.475972] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14890.502300] LNet: Added LNI 192.168.201.151@tcp13 [8/256/0/180] [14895.862139] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14895.892358] LNet: Added LNI 192.168.201.151@tcp14 [8/256/0/180] [14900.971892] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14900.995655] LNet: Added LNI 192.168.201.151@tcp15 [8/256/0/180] [14906.124437] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14906.152573] LNet: Added LNI 192.168.201.151@tcp16 [8/256/0/180] [14911.466904] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14911.490939] LNet: Added LNI 192.168.201.151@tcp17 [8/256/0/180] [14916.917651] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14916.939547] LNet: Added LNI 192.168.201.151@tcp18 [8/256/0/180] [14922.317174] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14922.359596] LNet: Added LNI 192.168.201.151@tcp19 [8/256/0/180] [14929.147244] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14929.216723] LNet: Added LNI 192.168.201.151@tcp20 [8/256/0/180] [14935.001839] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14935.030304] LNet: Added LNI 192.168.201.151@tcp21 [8/256/0/180] [14940.777280] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14940.821472] LNet: Added LNI 192.168.201.151@tcp22 [8/256/0/180] [14947.108297] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14947.134518] LNet: Added LNI 192.168.201.151@tcp23 [8/256/0/180] [14953.066977] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14953.121244] LNet: Added LNI 192.168.201.151@tcp24 [8/256/0/180] [14959.076119] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14959.116857] LNet: Added LNI 192.168.201.151@tcp25 [8/256/0/180] [14965.618847] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14965.656678] LNet: Added LNI 192.168.201.151@tcp26 [8/256/0/180] [14972.039490] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14972.082126] LNet: Added LNI 192.168.201.151@tcp27 [8/256/0/180] [14978.455236] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14978.478802] LNet: Added LNI 192.168.201.151@tcp28 [8/256/0/180] [14985.437170] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14985.476453] LNet: Added LNI 192.168.201.151@tcp29 [8/256/0/180] [14991.997408] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14992.039552] LNet: Added LNI 192.168.201.151@tcp30 [8/256/0/180] [14998.919677] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14998.992546] LNet: Added LNI 192.168.201.151@tcp31 [8/256/0/180] [15006.090323] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15006.128713] LNet: Added LNI 192.168.201.151@tcp32 [8/256/0/180] [15012.761637] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15012.792710] LNet: Added LNI 192.168.201.151@tcp33 [8/256/0/180] [15020.000423] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15020.030675] LNet: Added LNI 192.168.201.151@tcp34 [8/256/0/180] [15026.884984] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15026.929973] LNet: Added LNI 192.168.201.151@tcp35 [8/256/0/180] [15033.486176] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15033.507794] LNet: Added LNI 192.168.201.151@tcp36 [8/256/0/180] [15040.872719] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15040.897665] LNet: Added LNI 192.168.201.151@tcp37 [8/256/0/180] [15048.294184] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15048.326982] LNet: Added LNI 192.168.201.151@tcp38 [8/256/0/180] [15056.538776] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15056.575260] LNet: Added LNI 192.168.201.151@tcp39 [8/256/0/180] [15065.639667] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15065.674494] LNet: Added LNI 192.168.201.151@tcp40 [8/256/0/180] [15076.258535] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15076.294766] LNet: Added LNI 192.168.201.151@tcp41 [8/256/0/180] [15085.059379] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15085.080430] LNet: Added LNI 192.168.201.151@tcp42 [8/256/0/180] [15092.351677] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15092.416158] LNet: Added LNI 192.168.201.151@tcp43 [8/256/0/180] [15101.372279] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15101.392342] LNet: Added LNI 192.168.201.151@tcp44 [8/256/0/180] [15110.397515] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15110.450264] LNet: Added LNI 192.168.201.151@tcp45 [8/256/0/180] [15119.255657] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15119.304344] LNet: Added LNI 192.168.201.151@tcp46 [8/256/0/180] [15128.383513] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15128.405146] LNet: Added LNI 192.168.201.151@tcp47 [8/256/0/180] [15136.654648] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15136.688199] LNet: Added LNI 192.168.201.151@tcp48 [8/256/0/180] [15144.219886] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15144.252878] LNet: Added LNI 192.168.201.151@tcp49 [8/256/0/180] [15152.845262] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15152.904337] LNet: Added LNI 192.168.201.151@tcp50 [8/256/0/180] [15172.065106] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15172.072106] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15172.092506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15174.626236] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15174.627926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15174.643503] Lustre: Skipped 1 previous similar message [15174.655320] Lustre: Skipped 1 previous similar message [15178.028971] Lustre: server umount lustre-MDT0000 complete [15181.420723] LustreError: 473252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787130172 with bad export cookie 10703686642771136005 [15181.422399] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15181.439296] LustreError: 473252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15181.950296] Lustre: server umount lustre-MDT0001 complete [15199.920137] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [15200.224325] Lustre: 472533:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787130175/real 1787130175] req@00000000330c6fc4 x1873941392179456/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787130191 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15200.264422] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15202.273077] Lustre: 472532:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787130177/real 1787130177] req@000000000009c70d x1873941392179584/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787130193 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15205.344627] Lustre: 472531:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787130180/real 1787130180] req@000000004a7af3bf x1873941392179712/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787130196 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15207.395585] Lustre: 472530:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787130182/real 1787130182] req@00000000dcad3051 x1873941392179968/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787130198 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15208.416755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15213.536757] Lustre: 472532:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787130188/real 1787130188] req@00000000e542803e x1873941392180096/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787130204 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15226.912844] LustreError: 472529:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a83f76c5 x1873941392180864/t0(0) o250->MGC192.168.201.151@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15227.289917] LustreError: 475221:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15227.302983] LustreError: 475221:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [15227.350751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15229.413959] LustreError: 475222:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15229.425596] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15230.437370] LustreError: 475221:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15232.182407] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [15235.553424] LustreError: 475221:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15240.682527] LustreError: 475222:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15240.738820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15241.083770] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15245.159984] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15245.573145] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [15252.595050] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15258.792908] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15276.033670] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15276.043349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15276.045455] Lustre: Skipped 1 previous similar message [15276.050575] Lustre: Skipped 1 previous similar message [15281.124714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15281.130905] Lustre: Skipped 2 previous similar messages [15281.928436] Lustre: server umount lustre-OST0000 complete [15286.242418] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15286.259705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15286.268759] Lustre: Skipped 1 previous similar message [15292.646811] Lustre: server umount lustre-MDT0000 complete [15296.482253] LustreError: 501066:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15296.496915] LustreError: 501066:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15297.011091] LustreError: 473251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787130287 with bad export cookie 10703686642771138595 [15297.012272] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15297.018373] LustreError: 473251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [15297.274904] Lustre: server umount lustre-MDT0001 complete [15308.612712] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [15311.856116] Key type lgssc unregistered [15312.250403] LNet: 502262:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15312.258961] LNetError: 502262:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15313.312353] LNet: Removed LNI 192.168.201.151@tcp [15313.320255] LNetError: 502262:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15313.343122] LNet: Removed LNI 192.168.201.151@tcp10 [15313.345264] LNet: Removed LNI 192.168.201.151@tcp11 [15313.367614] LNet: Removed LNI 192.168.201.151@tcp12 [15313.371707] LNet: Removed LNI 192.168.201.151@tcp13 [15313.375653] LNet: Removed LNI 192.168.201.151@tcp14 [15313.379930] LNet: Removed LNI 192.168.201.151@tcp15 [15313.389302] LNet: Removed LNI 192.168.201.151@tcp16 [15313.402331] LNet: Removed LNI 192.168.201.151@tcp17 [15313.404360] LNet: Removed LNI 192.168.201.151@tcp18 [15313.424691] LNet: Removed LNI 192.168.201.151@tcp19 [15313.431505] LNet: Removed LNI 192.168.201.151@tcp20 [15313.439711] LNet: Removed LNI 192.168.201.151@tcp21 [15313.445116] LNet: Removed LNI 192.168.201.151@tcp22 [15313.449456] LNet: Removed LNI 192.168.201.151@tcp23 [15313.459731] LNet: Removed LNI 192.168.201.151@tcp24 [15313.462935] LNet: Removed LNI 192.168.201.151@tcp25 [15313.470610] LNet: Removed LNI 192.168.201.151@tcp26 [15313.482369] LNet: Removed LNI 192.168.201.151@tcp27 [15313.496365] LNet: Removed LNI 192.168.201.151@tcp28 [15313.505686] LNet: Removed LNI 192.168.201.151@tcp29 [15313.513424] LNet: Removed LNI 192.168.201.151@tcp30 [15313.528898] LNet: Removed LNI 192.168.201.151@tcp31 [15313.538337] LNet: Removed LNI 192.168.201.151@tcp32 [15313.545351] LNet: Removed LNI 192.168.201.151@tcp33 [15313.552638] LNet: Removed LNI 192.168.201.151@tcp34 [15313.558418] LNet: Removed LNI 192.168.201.151@tcp35 [15313.560738] LNet: Removed LNI 192.168.201.151@tcp36 [15313.566458] LNet: Removed LNI 192.168.201.151@tcp37 [15313.571403] LNet: Removed LNI 192.168.201.151@tcp38 [15313.578523] LNet: Removed LNI 192.168.201.151@tcp39 [15313.587525] LNet: Removed LNI 192.168.201.151@tcp40 [15313.592903] LNet: Removed LNI 192.168.201.151@tcp41 [15313.595846] LNet: Removed LNI 192.168.201.151@tcp42 [15313.598712] LNet: Removed LNI 192.168.201.151@tcp43 [15313.601762] LNet: Removed LNI 192.168.201.151@tcp44 [15313.607335] LNet: Removed LNI 192.168.201.151@tcp45 [15313.614767] LNet: Removed LNI 192.168.201.151@tcp46 [15313.618606] LNet: Removed LNI 192.168.201.151@tcp47 [15313.623428] LNet: Removed LNI 192.168.201.151@tcp48 [15313.626612] LNet: Removed LNI 192.168.201.151@tcp49 [15313.636236] LNet: Removed LNI 192.168.201.151@tcp50 [15314.574725] Key type .llcrypt unregistered [15314.577683] Key type ._llcrypt unregistered [15337.232978] Key type ._llcrypt registered [15337.236598] Key type .llcrypt registered [15337.370544] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [15352.642332] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [15354.160651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15354.189148] alg: No test for adler32 (adler32-zlib) [15355.308799] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [15355.818514] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [15357.528206] Key type lgssc registered [15358.586110] Lustre: Echo OBD driver; http://www.lustre.org/ [15394.651275] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15394.698582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15396.035493] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15396.087552] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15396.151124] Lustre: lustre-MDT0000: new disk, initializing [15396.223194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15396.238719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15401.082931] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [15412.525806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15412.706780] Lustre: 505458:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15412.773932] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15412.778281] Lustre: Skipped 1 previous similar message [15412.891975] Lustre: lustre-MDT0001: new disk, initializing [15413.035766] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15413.069890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15413.083217] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15417.158562] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [15425.570711] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15430.768924] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15437.111532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15437.287352] Lustre: *** cfs_fail_loc=90f, val=0*** [15437.339848] Lustre: lustre-OST0000: new disk, initializing [15437.344514] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15437.360774] Lustre: 507475:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15437.445353] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15442.963231] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [15442.974030] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15442.981878] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15443.129130] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15451.570510] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15458.274051] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15458.278373] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15458.286500] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15458.785565] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15458.796121] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15458.808317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15463.906709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15464.618795] Lustre: server umount lustre-OST0000 complete [15469.025511] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15469.028047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15469.048466] Lustre: Skipped 1 previous similar message [15469.070259] Lustre: Skipped 2 previous similar messages [15474.146739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15474.154712] Lustre: Skipped 1 previous similar message [15474.681563] Lustre: server umount lustre-MDT0000 complete [15478.901197] LustreError: 505450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787130469 with bad export cookie 16642757499221343572 [15478.906736] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15478.915168] LustreError: 505450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15479.282706] Lustre: server umount lustre-MDT0001 complete [15490.080796] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [15492.777883] Key type lgssc unregistered [15493.167696] LNet: 509380:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15493.180859] LNetError: 509380:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15493.217286] LNet: Removed LNI 192.168.201.151@tcp [15494.156961] Key type .llcrypt unregistered [15494.159421] Key type ._llcrypt unregistered [15512.573678] Key type ._llcrypt registered [15512.579274] Key type .llcrypt registered [15512.663580] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [15514.187380] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15514.217393] alg: No test for adler32 (adler32-zlib) [15515.376522] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [15515.694206] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [15517.392200] Key type lgssc registered [15518.649501] Lustre: Echo OBD driver; http://www.lustre.org/ [15525.755158] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15525.781719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15527.157258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15531.323615] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [15537.639766] LustreError: 510655:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15537.654409] LustreError: 510655:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [15540.692081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15541.003864] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15545.405513] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [15551.585441] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15557.720393] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15564.538659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15564.926486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15574.111678] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [15583.109577] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15595.668348] loop0: detected capacity change from 0 to 204800000 [15596.040451] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15596.062685] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15596.080102] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15603.009136] loop0: detected capacity change from 0 to 204800000 [15603.295396] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15603.324454] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15603.375863] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15609.487204] loop0: detected capacity change from 0 to 204800000 [15616.579809] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15616.720389] Lustre: fs15246-OST0000: new disk, initializing [15616.723628] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15616.729082] Lustre: 514250:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15616.780560] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15623.985364] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [15634.415868] loop1: detected capacity change from 0 to 204800000 [15639.970895] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15640.083417] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15640.126395] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15640.194296] Lustre: fs15246-MDT0000: new disk, initializing [15640.388375] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15640.420754] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15640.554090] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15641.635382] Lustre: 515372:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000a6d59ebb x1873942157508992/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787130648 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15645.467577] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [15650.736582] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15650.747045] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15650.814831] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15669.736212] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15669.757756] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15674.440450] Lustre: server umount fs15246-MDT0000 complete [15674.851739] LustreError: 510650:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15679.985830] LustreError: 513677:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15682.885390] Lustre: server umount fs15246-OST0000 complete [15694.310134] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15694.313799] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15694.338356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15695.334073] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15695.359321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15695.365055] Lustre: Skipped 1 previous similar message [15698.717721] Lustre: server umount lustre-OST0000 complete [15705.058364] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15705.065304] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15705.084673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15708.835768] Lustre: server umount lustre-MDT0000 complete [15710.690079] LustreError: 510655:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15710.715173] LustreError: 510655:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15712.420425] LustreError: 510637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787130703 with bad export cookie 11244008733676372157 [15712.426781] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15712.430843] LustreError: 510637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15712.631264] Lustre: server umount lustre-MDT0001 complete [15721.855571] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [15724.323244] Key type lgssc unregistered [15724.622672] LNet: 518467:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15724.628086] LNetError: 518467:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15724.658414] LNet: Removed LNI 192.168.201.151@tcp [15725.487696] Key type .llcrypt unregistered [15725.490063] Key type ._llcrypt unregistered [15746.753951] Key type ._llcrypt registered [15746.761172] Key type .llcrypt registered [15746.865877] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [15748.800975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15748.838907] alg: No test for adler32 (adler32-zlib) [15750.334285] Lustre: Lustre: Build Version: 2.17.56_3_g13af17f [15750.678026] LNet: Added LNI 192.168.201.151@tcp [8/256/0/180] [15752.400434] Key type lgssc registered [15754.543559] Lustre: Echo OBD driver; http://www.lustre.org/ [15794.368579] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [15804.250168] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [15836.170214] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15836.186493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15837.432634] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15837.474656] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15837.543317] Lustre: lustre-MDT0000: new disk, initializing [15837.632479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15837.659972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15842.898696] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [15852.075721] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15852.218638] Lustre: 522934:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15852.265185] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15852.268184] Lustre: Skipped 1 previous similar message [15852.319203] Lustre: lustre-MDT0001: new disk, initializing [15852.410654] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15852.455080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15852.472294] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15857.458908] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [15867.650731] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15873.223869] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15880.082763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15880.291871] Lustre: lustre-OST0000: new disk, initializing [15880.294971] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15880.300143] Lustre: 524900:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15880.381457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15886.991967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15887.002702] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15887.135267] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15887.307882] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_default_debug -1 all [15896.809971] Lustre: DEBUG MARKER: oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15901.468607] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15901.721874] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15906.341701] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15906.641409] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15912.929407] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15912.944630] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15912.945705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15912.964226] LustreError: Skipped 1 previous similar message [15915.058698] Lustre: server umount lustre-OST0000 complete [15919.074076] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15919.081108] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15919.092628] Lustre: Skipped 1 previous similar message [15919.099086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15919.106380] Lustre: Skipped 1 previous similar message [15923.171266] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15923.172309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15923.198551] Lustre: Skipped 1 previous similar message [15925.277025] Lustre: server umount lustre-MDT0000 complete [15928.972599] LustreError: 522925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787130919 with bad export cookie 15223047022967893213 [15928.976261] LustreError: MGC192.168.201.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15928.984336] LustreError: 522925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15929.264141] Lustre: server umount lustre-MDT0001 complete [15950.244750] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing set_hostid [15961.085479] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing load_modules_local [15993.579891] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15726 sec ========= 05:16:23 (1787130983) [15995.745968] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 05:16:25 (1787130985) === [16000.026378] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 05:16:29 (1787130989) === [16038.739576] Lustre: DEBUG MARKER: oleg151-server.virtnet: executing unload_modules_local [16041.286980] Key type lgssc unregistered [16041.575745] LNet: 531544:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16041.585201] LNetError: 531544:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16042.619072] LNet: Removed LNI 192.168.201.151@tcp [16043.560618] Key type .llcrypt unregistered [16043.562640] Key type ._llcrypt unregistered