[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 694707851 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2448MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.003363] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009015] pid_max: default: 32768 minimum: 301 [ 0.010148] LSM: Security Framework initializing [ 0.011061] Yama: becoming mindful. [ 0.012049] SELinux: Initializing. [ 0.013077] *** VALIDATE selinux *** [ 0.022034] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027340] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028155] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030096] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031117] *** VALIDATE tmpfs *** [ 0.032459] *** VALIDATE proc *** [ 0.034192] *** VALIDATE cgroup *** [ 0.035010] *** VALIDATE cgroup2 *** [ 0.036269] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037153] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039033] Spectre V2 : User space: Vulnerable [ 0.040012] Speculative Store Bypass: Vulnerable [ 0.043604] debug: unmapping init [mem 0xffffffffbb459000-0xffffffffbb460fff] [ 0.045241] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046772] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047022] ... version: 2 [ 0.048013] ... bit width: 48 [ 0.049012] ... generic registers: 4 [ 0.050012] ... value mask: 0000ffffffffffff [ 0.051012] ... max period: 00007fffffffffff [ 0.052014] ... fixed-purpose events: 3 [ 0.053012] ... event mask: 000000070000000f [ 0.054285] rcu: Hierarchical SRCU implementation. [ 0.056535] smp: Bringing up secondary CPUs ... [ 0.057649] x86: Booting SMP configuration: [ 0.058026] .... node #0, CPUs: #1 #2 #3 [ 0.066098] smp: Brought up 1 node, 4 CPUs [ 0.068013] smpboot: Max logical packages: 1 [ 0.069026] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.113312] node 0 deferred pages initialised in 42ms [ 0.117147] devtmpfs: initialized [ 0.118202] x86/mm: Memory block size: 128MB [ 0.120699] gcov: version magic: 0x41383552 [ 0.122025] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.123068] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.124220] pinctrl core: initialized pinctrl subsystem [ 0.125165] [ 0.125619] ************************************************************* [ 0.126011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.127011] ** ** [ 0.128011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.129010] ** ** [ 0.130009] ** This means that this kernel is built to expose internal ** [ 0.131011] ** IOMMU data structures, which may compromise security on ** [ 0.132009] ** your system. ** [ 0.133011] ** ** [ 0.134011] ** If you see this message and you are not debugging the ** [ 0.135013] ** kernel, report this immediately to your vendor! ** [ 0.136074] ** ** [ 0.137014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.138013] ************************************************************* [ 0.139712] NET: Registered protocol family 16 [ 0.140497] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.141062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.142056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.144033] cpuidle: using governor menu [ 0.145860] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.148425] PCI: Using configuration type 1 for base access [ 0.150129] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.158119] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.159018] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.161054] cryptd: max_cpu_qlen set to 1000 [ 0.163266] ACPI: Added _OSI(Module Device) [ 0.164013] ACPI: Added _OSI(Processor Device) [ 0.165011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.166012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.170606] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.172875] ACPI: Interpreter enabled [ 0.173060] ACPI: PM: (supports S0 S3 S4 S5) [ 0.174011] ACPI: Using IOAPIC for interrupt routing [ 0.175109] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.176411] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.186770] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.187036] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.188019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.189083] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.191496] acpiphp: Slot [2] registered [ 0.192134] acpiphp: Slot [5] registered [ 0.193124] acpiphp: Slot [6] registered [ 0.194168] acpiphp: Slot [7] registered [ 0.195102] acpiphp: Slot [8] registered [ 0.196100] acpiphp: Slot [9] registered [ 0.197202] acpiphp: Slot [10] registered [ 0.198171] acpiphp: Slot [3] registered [ 0.199089] acpiphp: Slot [4] registered [ 0.200099] acpiphp: Slot [11] registered [ 0.201088] acpiphp: Slot [12] registered [ 0.202099] acpiphp: Slot [13] registered [ 0.203106] acpiphp: Slot [14] registered [ 0.204105] acpiphp: Slot [15] registered [ 0.205112] acpiphp: Slot [16] registered [ 0.206101] acpiphp: Slot [17] registered [ 0.207092] acpiphp: Slot [18] registered [ 0.208093] acpiphp: Slot [19] registered [ 0.209101] acpiphp: Slot [20] registered [ 0.210147] acpiphp: Slot [21] registered [ 0.211126] acpiphp: Slot [22] registered [ 0.212089] acpiphp: Slot [23] registered [ 0.213164] acpiphp: Slot [24] registered [ 0.214100] acpiphp: Slot [25] registered [ 0.215142] acpiphp: Slot [26] registered [ 0.216168] acpiphp: Slot [27] registered [ 0.217168] acpiphp: Slot [28] registered [ 0.218118] acpiphp: Slot [29] registered [ 0.219162] acpiphp: Slot [30] registered [ 0.220144] acpiphp: Slot [31] registered [ 0.221052] PCI host bridge to bus 0000:00 [ 0.222017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.223023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.224019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.225021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.226026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.227026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.228167] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.230063] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.232334] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.238014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.242015] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.243021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.244016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.245016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.246626] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.247815] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.248044] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.249850] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.251914] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.260015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.263016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.267120] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.270019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.273016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.280017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.286314] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.289021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.292014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.299017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.308272] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.322015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.333014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.356014] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.365000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.372018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.379018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.407016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.425516] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.517014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.525017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.552000] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.562165] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.573017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.584016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.617014] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.633407] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.635370] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.638430] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.640349] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.643212] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.647117] iommu: Default domain type: Passthrough [ 0.648558] SCSI subsystem initialized [ 0.650423] ACPI: bus type USB registered [ 0.652218] usbcore: registered new interface driver usbfs [ 0.654157] usbcore: registered new interface driver hub [ 0.656183] usbcore: registered new device driver usb [ 0.659190] pps_core: LinuxPPS API ver. 1 registered [ 0.661010] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.664061] PTP clock support registered [ 0.667089] EDAC MC: Ver: 3.0.0 [ 0.668477] PCI: Using ACPI for IRQ routing [ 0.669633] NetLabel: Initializing [ 0.671012] NetLabel: domain hash size = 128 [ 0.672051] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.675076] NetLabel: unlabeled traffic allowed by default [ 0.677150] vgaarb: loaded [ 0.679295] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.681014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.688000] clocksource: Switched to clocksource kvm-clock [ 0.796519] VFS: Disk quotas dquot_6.6.0 [ 0.798038] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.800413] *** VALIDATE ramfs *** [ 0.801588] *** VALIDATE hugetlbfs *** [ 0.803143] pnp: PnP ACPI init [ 0.806111] pnp: PnP ACPI: found 6 devices [ 0.835701] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.838969] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.841395] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.843571] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.845946] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.848304] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.851252] NET: Registered protocol family 2 [ 0.853842] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.858765] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.862429] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.867968] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.871718] TCP: Hash tables configured (established 65536 bind 65536) [ 0.874616] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.877820] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.880683] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.883825] NET: Registered protocol family 1 [ 0.886835] RPC: Registered named UNIX socket transport module. [ 0.889113] RPC: Registered udp transport module. [ 0.890687] RPC: Registered tcp transport module. [ 0.892324] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.894801] NET: Registered protocol family 44 [ 0.896436] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.898898] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.901080] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.903771] PCI: CLS 0 bytes, default 64 [ 0.905444] Unpacking initramfs... [ 2.430526] debug: unmapping init [mem 0xffff8ea3fcc54000-0xffff8ea3fffbffff] [ 2.435167] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.437464] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.440195] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.983119] Initialise system trusted keyrings [ 2.984351] Key type blacklist registered [ 2.985957] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.994351] zbud: loaded [ 2.997269] *** VALIDATE nfs *** [ 2.998456] *** VALIDATE nfs4 *** [ 3.000270] pstore: using deflate compression [ 3.004510] Platform Keyring initialized [ 3.109861] NET: Registered protocol family 38 [ 3.112148] Key type asymmetric registered [ 3.114370] Asymmetric key parser 'x509' registered [ 3.116191] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.119037] io scheduler mq-deadline registered [ 3.120734] io scheduler kyber registered [ 3.122406] io scheduler bfq registered [ 3.124089] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.127241] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.129983] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.132885] ACPI: Power Button [PWRF] [ 3.141965] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.149141] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.162962] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.178814] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.197279] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.224864] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.254718] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.260101] Non-volatile memory driver v1.3 [ 3.261972] Linux agpgart interface v0.103 [ 3.290566] virtio_blk virtio1: [vda] 139288 512-byte logical blocks (71.3 MB/68.0 MiB) [ 3.293681] vda: detected capacity change from 0 to 71315456 [ 3.345390] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.347780] vdb: detected capacity change from 0 to 1073741824 [ 3.372716] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.375135] vdc: detected capacity change from 0 to 2621440000 [ 3.391730] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.394463] vdd: detected capacity change from 0 to 2621440000 [ 3.410402] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.414274] vde: detected capacity change from 0 to 4294967296 [ 3.434309] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.437841] vdf: detected capacity change from 0 to 4294967296 [ 3.450450] libphy: Fixed MDIO Bus: probed [ 3.456896] usbcore: registered new interface driver usbserial_generic [ 3.459714] usbserial: USB Serial support registered for generic [ 3.462297] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.467380] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.469126] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.471644] mousedev: PS/2 mouse device common for all mice [ 3.475182] rtc_cmos 00:05: RTC can wake from S4 [ 3.481108] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.483218] rtc_cmos 00:05: registered as rtc0 [ 3.488237] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.491703] intel_pstate: CPU model not supported [ 3.494740] hid: raw HID events driver (C) Jiri Kosina [ 3.497499] usbcore: registered new interface driver usbhid [ 3.498785] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.500202] usbhid: USB HID core driver [ 3.500350] drop_monitor: Initializing network drop monitor service [ 3.500471] Initializing XFRM netlink socket [ 3.500897] NET: Registered protocol family 10 [ 3.503433] Segment Routing with IPv6 [ 3.514940] NET: Registered protocol family 17 [ 3.516459] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.518410] mpls_gso: MPLS GSO support [ 3.526973] RAS: Correctable Errors collector initialized. [ 3.528695] AVX version of gcm_enc/dec engaged. [ 3.530007] AES CTR mode by8 optimization enabled [ 3.606078] sched_clock: Marking stable (3606062166, 0)->(4878755280, -1272693114) [ 3.609103] registered taskstats version 1 [ 3.610790] Loading compiled-in X.509 certificates [ 3.612560] zswap: loaded using pool lzo/zbud [ 3.640254] Key type big_key registered [ 3.655841] Key type encrypted registered [ 3.657638] ima: No TPM chip found, activating TPM-bypass! [ 3.659990] ima: Allocated hash algorithm: sha1 [ 3.662417] ima: No architecture policies found [ 3.664447] evm: Initialising EVM extended attributes: [ 3.666783] evm: security.selinux [ 3.668403] evm: security.ima [ 3.670132] evm: security.capability [ 3.671392] evm: HMAC attrs: 0x1 [ 3.674274] rtc_cmos 00:05: setting system clock to 2026-06-17 07:44:00 UTC (1781682240) [ 3.682490] debug: unmapping init [mem 0xffffffffbc403000-0xffffffffbc5fffff] [ 3.686172] debug: unmapping init [mem 0xffffffffbb182000-0xffffffffbb458fff] [ 3.695177] Write protecting the kernel read-only data: 28672k [ 3.698930] debug: unmapping init [mem 0xffffffffb9803000-0xffffffffb99fffff] [ 3.701937] debug: unmapping init [mem 0xffffffffba114000-0xffffffffba1fffff] [ 3.744372] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.759693] systemd[1]: Detected virtualization kvm. [ 3.762699] systemd[1]: Detected architecture x86-64. [ 3.765130] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.797985] systemd[1]: No hostname configured. [ 3.800852] systemd[1]: Set hostname to . [ 3.803869] random: systemd: uninitialized urandom read (16 bytes read) [ 3.807827] systemd[1]: Initializing machine ID from random generator. [ 3.865412] random: ln: uninitialized urandom read (6 bytes read) [ 3.981109] random: systemd: uninitialized urandom read (16 bytes read) [ 3.984502] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.993747] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ 4.011710] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Swap. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Journal Service... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.757755] device-mapper: uevent: version 1.0.3 [ 4.760403] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ 5.932318] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ 5.980175] random: fast init done [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 6.202418] scsi host0: ata_piix [ 6.227147] scsi host1: ata_piix [ 6.228712] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 6.231647] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 12.230885] random: crng init done [ 12.232328] random: 7 urandom warning(s) missed due to ratelimiting [ 13.634400] dracut-initqueue[586]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 15.019039] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 16.779866] printk: systemd: 25 output lines suppressed due to ratelimiting [ 17.178613] SELinux: Disabled at runtime. [ 17.241187] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 17.251418] systemd[1]: Detected virtualization kvm. [ 17.253486] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 18.505658] systemd[1]: initrd-switch-root.service: Succeeded. [ 18.509933] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 18.518872] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 18.523722] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 18.527495] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 18.587119] systemd[1]: Starting Journal Service... Starting Journal Service... [ 18.599996] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. Starting Remount Root and Kernel File Systems... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd Root File System. Mounting POSIX Message Queue File System... Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File Syste[ 18.815406] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS m... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ 19.617580] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 20.464548] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 20.496459] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 21.857492] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 21.959557] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit)[ 25.592848] Key type dns_resolver registered [** ] A start job is running for Configur…-only root support (7s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 26.804741] NFS: Registering the id_resolver key type [ 26.808970] Key type id_resolver registered [ 26.811229] Key type id_legacy registered [ *** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started dnf makecache --timer. Starting Restore /run/initramfs on shutdown... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Hostname Service... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Login Service. [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg444-server login: [ 85.429044] hrtimer: interrupt took 2182365 ns [ 96.672751] libcfs: loading out-of-tree module taints kernel. [ 96.757171] Key type ._llcrypt registered [ 96.759033] Key type .llcrypt registered [ 96.955827] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 122.419982] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 124.036477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 124.063407] alg: No test for adler32 (adler32-zlib) [ 125.638535] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 126.681175] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 128.519143] Key type lgssc registered [ 130.601191] Lustre: Echo OBD driver; http://www.lustre.org/ [ 147.679802] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 192.282237] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 206.635368] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 206.683522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 208.040880] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 208.129899] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 208.210497] Lustre: lustre-MDT0000: new disk, initializing [ 208.314359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 208.352386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 213.426569] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 227.044612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 227.160742] Lustre: 6529:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 227.216983] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 227.221809] Lustre: Skipped 1 previous similar message [ 227.355623] Lustre: lustre-MDT0001: new disk, initializing [ 227.428553] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 227.462198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 227.479465] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 231.921404] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 236.453406] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 247.518566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 247.750296] Lustre: lustre-OST0000: new disk, initializing [ 247.753804] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 247.758612] Lustre: 8468:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 247.805283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 253.738119] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 253.972922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 253.986785] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 254.082280] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 269.142388] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 269.307877] Lustre: lustre-OST0001: new disk, initializing [ 269.310404] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 269.315702] Lustre: 9542:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 269.379233] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 275.552063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 275.564085] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 275.636028] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 275.866809] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 288.178019] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 299.143205] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 306.182669] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing check_logdir /tmp/testlogs/ [ 313.060992] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing yml_node [ 318.358674] Lustre: DEBUG MARKER: Client: 2.17.54.40 [ 321.108557] Lustre: DEBUG MARKER: MDS: 2.17.54.40 [ 323.607058] Lustre: DEBUG MARKER: OSS: 2.17.54.40 [ 324.996188] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Jun 17 03:49:20 EDT 2026 [ 342.910625] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 344.665626] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 362.979214] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 362.984241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 363.006374] Lustre: Skipped 1 previous similar message [ 363.025087] Lustre: Skipped 3 previous similar messages [ 367.820973] Lustre: server umount lustre-MDT0000 complete [ 372.805725] LustreError: 9539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781682609 with bad export cookie 14248501067814436078 [ 372.813712] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 372.825391] LustreError: 9539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 373.215917] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 373.216991] LustreError: 8065:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 373.237324] Lustre: Skipped 2 previous similar messages [ 373.265090] LustreError: 8065:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 373.273245] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 373.289219] Lustre: Skipped 1 previous similar message [ 377.247778] LustreError: 6537:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 377.253142] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 377.267857] LustreError: 6537:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 377.284524] Lustre: Skipped 1 previous similar message [ 379.862444] Lustre: server umount lustre-MDT0001 complete [ 386.176421] Lustre: server umount lustre-OST0000 complete [ 392.569279] Lustre: server umount lustre-OST0001 complete [ 405.740304] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 420.837241] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 453.861963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 454.063636] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 454.090477] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 454.155446] Lustre: lustre-MDT0000: new disk, initializing [ 454.215461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 454.229467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 457.419802] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 465.311061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 465.377945] Lustre: 15113:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 465.424854] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 465.428823] Lustre: Skipped 1 previous similar message [ 465.491423] Lustre: lustre-MDT0001: new disk, initializing [ 465.580139] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 465.617899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 465.624973] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 468.883468] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 476.945518] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 481.795629] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 487.164617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 487.371299] Lustre: lustre-OST0000: new disk, initializing [ 487.375227] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 487.383299] Lustre: 17081:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 487.471719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 489.010569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 489.017755] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 489.130098] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 491.841060] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 498.692264] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 501.557590] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 501.669541] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 504.389460] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 504.505725] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 509.926322] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 509.926382] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 509.927761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 509.937584] Lustre: Skipped 2 previous similar messages [ 511.797790] Lustre: server umount lustre-OST0000 complete [ 515.041852] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 515.050409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 515.054764] Lustre: Skipped 2 previous similar messages [ 520.358802] Lustre: server umount lustre-MDT0000 complete [ 522.677832] LustreError: 15105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781682759 with bad export cookie 14248501067814437611 [ 522.681346] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 522.686060] LustreError: 15105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 522.898516] Lustre: server umount lustre-MDT0001 complete [ 525.353573] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 03:52:41 (1781682761) [ 526.942896] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 528.052732] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 03:52:43 (1781682763) [ 532.053865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 532.382951] LustreError: 19490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 532.451953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 535.014703] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 537.569135] LustreError: 19491:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 539.880307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 542.476408] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 546.919650] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 550.266658] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 554.106262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 557.549921] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 561.490945] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 569.826097] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 569.829060] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 569.829724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 569.829729] Lustre: Skipped 2 previous similar messages [ 569.832215] LustreError: Skipped 1 previous similar message [ 569.852094] Lustre: Skipped 2 previous similar messages [ 571.550588] Lustre: server umount lustre-OST0000 complete [ 579.705969] Lustre: server umount lustre-MDT0000 complete [ 580.064439] LustreError: 20920:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 580.079253] LustreError: 20920:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 581.771561] LustreError: 21982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781682818 with bad export cookie 14248501067814438493 [ 581.777519] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 581.780110] LustreError: 21982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 581.981654] Lustre: server umount lustre-MDT0001 complete [ 588.092460] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 589.683500] Key type lgssc unregistered [ 589.844072] LNet: 23020:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 589.849057] LNetError: 23020:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 589.858996] LNet: Removed LNI 192.168.204.144@tcp [ 590.347159] Key type .llcrypt unregistered [ 590.349798] Key type ._llcrypt unregistered [ 603.139217] Key type ._llcrypt registered [ 603.140940] Key type .llcrypt registered [ 603.201826] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 603.674779] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 603.746634] alg: No test for adler32 (adler32-zlib) [ 604.699682] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 604.851667] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 606.463169] Key type lgssc registered [ 607.031018] Lustre: Echo OBD driver; http://www.lustre.org/ [ 611.411545] loop: module loaded [ 611.449458] loop0: detected capacity change from 0 to 204800000 [ 611.567763] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 611.574859] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 611.794043] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 615.023341] loop0: detected capacity change from 0 to 204800000 [ 615.114250] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 615.123609] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 615.130307] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 617.786196] loop0: detected capacity change from 0 to 204800000 [ 620.577506] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 620.591060] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 621.742327] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 621.779568] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 621.835293] Lustre: lustre-MDT0000: new disk, initializing [ 621.888180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 621.899480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 623.874440] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 628.444364] loop1: detected capacity change from 0 to 204800000 [ 630.939636] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 631.068635] Lustre: lustre-OST07c6: new disk, initializing [ 631.070835] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 631.073354] Lustre: Skipped 1 previous similar message [ 631.075956] Lustre: 26184:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 631.110283] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 633.977434] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 635.405672] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 635.411741] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 635.435757] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 637.589611] Lustre: 26192:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 637.594152] Lustre: 26192:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 637.598535] Lustre: 26192:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 637.602968] Lustre: 26192:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 637.608599] Lustre: 26192:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 637.615831] Lustre: 26192:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 652.529440] Lustre: 25169:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 652.534608] Lustre: 25169:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 652.539555] Lustre: 25169:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 652.544195] Lustre: 25169:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 652.549068] Lustre: 25169:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 652.552990] Lustre: 25169:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 656.419522] Lustre: 25171:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 656.424088] Lustre: 25171:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 656.427857] Lustre: 25171:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 656.431504] Lustre: 25171:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 656.435524] Lustre: 25171:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 656.440511] Lustre: 25171:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 656.444865] Lustre: 25171:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 656.449119] Lustre: 25171:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 656.453230] Lustre: 25171:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 656.456667] Lustre: 25171:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 656.461507] Lustre: 25171:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 656.465392] Lustre: 25171:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 659.563072] Lustre: 26192:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 659.569672] Lustre: 26192:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 659.574516] Lustre: 26192:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 659.578683] Lustre: 26192:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 659.583866] Lustre: 26192:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 659.589728] Lustre: 26192:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 659.594700] Lustre: 26192:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 659.599423] Lustre: 26192:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 659.604558] Lustre: 26192:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 659.609784] Lustre: 26192:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 659.614817] Lustre: 26192:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 659.619297] Lustre: 26192:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 666.473146] Lustre: 26192:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 666.477171] Lustre: 26192:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 666.481032] Lustre: 26192:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 666.486920] Lustre: 26192:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 666.491187] Lustre: 26192:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 666.495261] Lustre: 26192:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 666.499074] Lustre: 26192:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 666.503536] Lustre: 26192:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 666.512341] Lustre: 26192:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 666.516784] Lustre: 26192:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 666.521685] Lustre: 26192:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 666.525921] Lustre: 26192:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 676.602173] Lustre: 25171:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 676.607452] Lustre: 25171:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 676.614499] Lustre: 25171:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 676.620384] Lustre: 25171:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 676.628302] Lustre: 25171:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 676.633973] Lustre: 25171:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 676.637576] Lustre: 25171:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 676.642184] Lustre: 25171:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 676.645904] Lustre: 25171:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 676.649741] Lustre: 25171:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 676.653905] Lustre: 25171:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 676.657356] Lustre: 25171:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 720.466710] Lustre: server umount lustre-OST07c6 complete [ 733.707873] Lustre: server umount lustre-MDT0000 complete [ 739.099764] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 740.199442] Key type lgssc unregistered [ 740.349586] LNet: 28740:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 740.352144] LNetError: 28740:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 740.360430] LNet: Removed LNI 192.168.204.144@tcp [ 740.675675] Key type .llcrypt unregistered [ 740.677099] Key type ._llcrypt unregistered [ 748.732170] Key type ._llcrypt registered [ 748.733675] Key type .llcrypt registered [ 748.774468] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 749.121855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 749.147187] alg: No test for adler32 (adler32-zlib) [ 750.031764] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 750.134350] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 751.735170] Key type lgssc registered [ 752.444427] Lustre: Echo OBD driver; http://www.lustre.org/ [ 757.931693] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 757.943807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 759.327671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 762.754567] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 768.632107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 768.848965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 772.605446] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 779.060042] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 784.627356] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 792.791613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 793.394733] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 800.464753] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 805.689743] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 808.187499] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 819.171698] LustreError: lustre-OST0000-osc-MDT0000: operation ost_setattr to node 0@lo failed: rc = -107 [ 819.175357] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 819.183732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 819.425289] Lustre: server umount lustre-OST0000 complete [ 821.216485] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 821.218061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 821.223437] Lustre: Skipped 1 previous similar message [ 821.229686] Lustre: Skipped 1 previous similar message [ 826.336675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 826.340665] Lustre: Skipped 1 previous similar message [ 827.256797] Lustre: server umount lustre-MDT0000 complete [ 828.725239] LustreError: 29971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781683065 with bad export cookie 10571606456906130144 [ 828.728280] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 828.729546] LustreError: 29971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 839.191643] Lustre: server umount lustre-MDT0001 complete [ 843.403711] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 844.753128] Key type lgssc unregistered [ 844.926816] LNet: 33722:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 844.931502] LNetError: 33722:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 844.941743] LNet: Removed LNI 192.168.204.144@tcp [ 845.397615] Key type .llcrypt unregistered [ 845.399376] Key type ._llcrypt unregistered [ 854.477957] Key type ._llcrypt registered [ 854.479753] Key type .llcrypt registered [ 854.539464] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 854.994353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 855.010553] alg: No test for adler32 (adler32-zlib) [ 855.890233] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 856.009397] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 857.623146] Key type lgssc registered [ 858.198245] Lustre: Echo OBD driver; http://www.lustre.org/ [ 862.014453] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 862.028609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 863.274095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 865.229966] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 868.919821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 869.072048] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 870.684345] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 873.342687] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 875.350648] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 877.697647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 877.839496] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 879.963287] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 882.535594] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 889.126871] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 899.553272] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 899.554403] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 899.556137] LustreError: Skipped 1 previous similar message [ 899.556658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 899.560498] Lustre: Skipped 1 previous similar message [ 904.334946] Lustre: server umount lustre-OST0000 complete [ 909.793844] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 909.794800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 909.806027] Lustre: Skipped 1 previous similar message [ 909.809038] Lustre: Skipped 1 previous similar message [ 911.780131] Lustre: server umount lustre-MDT0000 complete [ 913.467363] LustreError: 34952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781683150 with bad export cookie 18290020317217154173 [ 913.471359] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 913.472572] LustreError: 34952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 913.651657] Lustre: server umount lustre-MDT0001 complete [ 917.791369] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 919.031075] Key type lgssc unregistered [ 919.183494] LNet: 38524:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 919.186582] LNetError: 38524:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 919.194318] LNet: Removed LNI 192.168.204.144@tcp [ 919.509653] Key type .llcrypt unregistered [ 919.511740] Key type ._llcrypt unregistered [ 929.234949] Key type ._llcrypt registered [ 929.236728] Key type .llcrypt registered [ 929.282938] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 929.809835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 929.829203] alg: No test for adler32 (adler32-zlib) [ 930.689812] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 930.770871] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 932.359154] Key type lgssc registered [ 932.780146] Lustre: Echo OBD driver; http://www.lustre.org/ [ 935.600720] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 935.607164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 936.767565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 938.351881] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 941.599022] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 941.736205] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 943.108986] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 945.630496] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 947.507971] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 949.827634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 949.940775] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 951.920233] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 954.645200] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 961.195908] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 970.515584] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.44@tcp (stopping) [ 971.747693] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 971.748981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 971.751662] Lustre: Skipped 2 previous similar messages [ 971.755163] Lustre: Skipped 2 previous similar messages [ 974.049137] Lustre: server umount lustre-MDT0000 complete [ 975.572372] LustreError: 40653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781683212 with bad export cookie 4096699574386652690 [ 975.576039] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 975.577710] LustreError: 40653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 975.625514] LustreError: 42405:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 975.635349] LustreError: 42405:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 975.636431] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.44@tcp (stopping) [ 976.864141] LustreError: 39942:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 976.864379] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 980.745769] LustreError: 42405:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 980.746137] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.44@tcp (stopping) [ 980.754113] Lustre: Skipped 1 previous similar message [ 981.742101] Lustre: server umount lustre-MDT0001 complete [ 1008.095146] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1008.161985] Lustre: server umount lustre-OST0000 complete [ 1014.289871] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 1015.391246] Key type lgssc unregistered [ 1015.528432] LNet: 43796:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1015.531231] LNetError: 43796:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1015.544260] LNet: Removed LNI 192.168.204.144@tcp [ 1015.895863] Key type .llcrypt unregistered [ 1015.897725] Key type ._llcrypt unregistered [ 1025.198176] Key type ._llcrypt registered [ 1025.199859] Key type .llcrypt registered [ 1025.235843] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 1025.698195] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1025.718976] alg: No test for adler32 (adler32-zlib) [ 1026.576795] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 1026.654037] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 1028.231130] Key type lgssc registered [ 1028.597764] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1031.263154] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1031.271152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1032.426927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1033.852040] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1036.591297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1036.710187] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1038.010775] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1040.335182] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1042.229745] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1044.559339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1044.680583] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1046.860357] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1049.746233] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1058.024056] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1063.392815] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1063.393588] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1063.395743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1063.403384] Lustre: Skipped 1 previous similar message [ 1065.234317] Lustre: lustre-OST0000: Not available for connect from 192.168.204.44@tcp (stopping) [ 1065.236773] Lustre: Skipped 1 previous similar message [ 1068.503615] Lustre: server umount lustre-OST0000 complete [ 1081.659761] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1081.660730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1081.665096] Lustre: Skipped 1 previous similar message [ 1083.744565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1083.747457] Lustre: Skipped 2 previous similar messages [ 1087.847331] Lustre: server umount lustre-MDT0000 complete [ 1088.992243] LustreError: 45940:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1089.000262] LustreError: 45940:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1089.371409] LustreError: 45184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781683326 with bad export cookie 1874870995081011704 [ 1089.374797] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1089.377488] LustreError: 45184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1095.563697] Lustre: server umount lustre-MDT0001 complete [ 1099.816337] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 1101.058544] Key type lgssc unregistered [ 1101.212623] LNet: 48926:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1101.216106] LNetError: 48926:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1101.223412] LNet: Removed LNI 192.168.204.144@tcp [ 1101.572329] Key type .llcrypt unregistered [ 1101.573713] Key type ._llcrypt unregistered [ 1111.476078] Key type ._llcrypt registered [ 1111.477376] Key type .llcrypt registered [ 1111.523810] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 1111.978172] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1112.002296] alg: No test for adler32 (adler32-zlib) [ 1112.860192] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 1112.953850] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 1114.535129] Key type lgssc registered [ 1114.929255] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1117.766875] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1117.774340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1118.939832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1120.520667] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1123.769665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1123.909095] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1125.366553] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1128.138310] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1130.128515] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1132.736711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1132.892124] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1134.984130] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1137.430483] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1145.190171] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1150.330311] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1151.609924] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1162.375208] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-06-17 4h03m18s [ 1162.944374] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1164.768175] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1164.771837] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1164.776097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1165.793105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1165.796197] Lustre: Skipped 4 previous similar messages [ 1168.656349] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.44@tcp (stopping) [ 1169.814742] Lustre: server umount lustre-MDT0000 complete [ 1170.913695] LustreError: 51761:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1170.918567] LustreError: 51761:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1171.026868] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1171.936095] LustreError: 50321:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1171.944459] LustreError: 50321:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1173.342154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1173.397448] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1173.492794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1173.529216] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1174.500376] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1174.953791] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1176.160691] Lustre: DEBUG MARKER: Wait for df (36487) ... [ 1176.741293] Lustre: DEBUG MARKER: done [ 1178.591871] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1178.595297] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1178.600137] Lustre: Skipped 2 previous similar messages [ 1178.602548] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1178.605347] Lustre: Skipped 1 previous similar message [ 1179.619471] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1179.924092] Lustre: server umount lustre-OST0000 complete [ 1183.387437] Lustre: server umount lustre-MDT0000 complete [ 1184.659700] LustreError: 50300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781683421 with bad export cookie 10088022732967742207 [ 1184.663654] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1184.665227] LustreError: 50300:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1190.823475] Lustre: server umount lustre-MDT0001 complete [ 1194.576252] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 1195.605370] Key type lgssc unregistered [ 1195.729411] LNet: 55253:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1195.733185] LNetError: 55253:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1195.746307] LNet: Removed LNI 192.168.204.144@tcp [ 1196.049486] Key type .llcrypt unregistered [ 1196.050943] Key type ._llcrypt unregistered [ 1210.925244] Key type ._llcrypt registered [ 1210.926316] Key type .llcrypt registered [ 1210.964730] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 1211.359460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1211.423503] alg: No test for adler32 (adler32-zlib) [ 1212.290239] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 1212.376555] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 1213.967693] Key type lgssc registered [ 1214.373934] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1217.092871] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1217.098764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1218.201774] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1218.212581] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1218.277545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1219.685367] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1222.757404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1222.800633] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1222.811349] Lustre: 57438:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1222.910714] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1224.397374] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1226.827635] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1228.741847] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1231.212527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1231.283766] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1231.355764] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1233.049663] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1233.387462] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1236.001662] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1239.457362] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-06-17 4h04m35s [ 1240.061990] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1242.543671] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1243.399526] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1243.416109] LustreError: 59305:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1263.897655] Lustre: lustre-MDT0000: Client 6f00a43b-776e-455e-a390-8580afc73613 (at 192.168.204.44@tcp) reconnecting [ 1283.463119] LustreError: 59305:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1284.277465] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1285.193162] Lustre: DEBUG MARKER: done [ 1289.185503] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1289.189096] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1289.195551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1289.697899] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1289.700088] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1289.704525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1292.945574] Lustre: server umount lustre-OST0000 complete [ 1294.816530] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1294.816583] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1294.817101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1294.825288] Lustre: Skipped 1 previous similar message [ 1300.314656] Lustre: server umount lustre-MDT0000 complete [ 1301.702206] LustreError: 57428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781683538 with bad export cookie 9956854275453036686 [ 1301.704789] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1301.707656] LustreError: 57428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1301.829560] Lustre: server umount lustre-MDT0001 complete [ 1305.570787] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 1306.566356] Key type lgssc unregistered [ 1306.691366] LNet: 61758:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1306.694983] LNetError: 61758:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1306.702369] LNet: Removed LNI 192.168.204.144@tcp [ 1306.994282] Key type .llcrypt unregistered [ 1306.996035] Key type ._llcrypt unregistered [ 1322.899921] Key type ._llcrypt registered [ 1322.901153] Key type .llcrypt registered [ 1322.939038] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 1323.257943] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1323.264524] alg: No test for adler32 (adler32-zlib) [ 1324.115281] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 1324.201826] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 1325.783162] Key type lgssc registered [ 1326.152857] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1327.278453] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1327.284488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1328.372386] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1328.380166] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1328.439920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1328.762700] Lustre: Failing over lustre-MDT0000 [ 1328.865691] Lustre: server umount lustre-MDT0000 complete [ 1331.051443] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 04:06:07 (1781683567) [ 1333.304165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1333.449883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1334.776529] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1337.751174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1337.784878] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1337.797459] Lustre: 64531:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1337.904906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1339.316382] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1341.749399] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1343.549057] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1345.772831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1345.846177] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1345.907620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1347.924850] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1349.129881] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1350.326638] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1354.474524] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1358.303835] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1358.307185] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1358.312952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1359.328931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1359.329066] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1359.331547] Lustre: Skipped 1 previous similar message [ 1359.337373] Lustre: Skipped 1 previous similar message [ 1362.527285] Lustre: server umount lustre-MDT0000 complete [ 1363.813036] LustreError: 64526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781683600 with bad export cookie 1668758758276145811 [ 1363.815888] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1363.817208] LustreError: 64526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1363.922122] Lustre: server umount lustre-MDT0001 complete [ 1365.050266] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1366.561792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1369.382036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1373.599687] LustreError: 63454:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ca8d3854 x1868230621690880/t0(0) o250->MGC192.168.204.144@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1373.732984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1373.766518] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1375.191344] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1378.072645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1378.167205] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1379.238761] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1379.591789] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1380.255131] Lustre: 63458:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781683601/real 1781683601] req@00000000286f1790 x1868230621690496/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781683617 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1382.101980] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1384.053694] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1385.439162] Lustre: 63455:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781683605/real 1781683605] req@0000000030505f36 x1868230621690624/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781683621 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1385.448312] Lustre: 63455:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1388.512323] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1388.515108] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1388.520145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1388.522962] Lustre: Skipped 1 previous similar message [ 1390.559372] Lustre: 63457:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781683610/real 1781683610] req@0000000054808183 x1868230621691008/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781683626 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1392.023044] Lustre: server umount lustre-MDT0000 complete [ 1393.361138] LustreError: 64524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781683630 with bad export cookie 1668758758276148079 [ 1393.365472] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1393.366213] LustreError: 64524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1393.476660] Lustre: server umount lustre-MDT0001 complete [ 1395.854476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1399.222044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1403.807581] LustreError: 63454:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000020e8572f x1868230621710848/t0(0) o250->MGC192.168.204.144@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1403.912904] LustreError: 66379:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1403.919372] LustreError: 66379:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1403.935228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1403.937731] Lustre: Skipped 1 previous similar message [ 1403.956112] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1405.365212] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1408.238025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1408.341269] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1408.347078] Lustre: Skipped 2 previous similar messages [ 1409.444034] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1409.447986] Lustre: Skipped 1 previous similar message [ 1409.806602] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1410.464088] Lustre: 63457:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781683631/real 1781683631] req@00000000136d06df x1868230621710464/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781683647 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1412.249256] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1414.027029] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1414.623168] Lustre: 63455:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781683635/real 1781683635] req@0000000054321978 x1868230621710592/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781683651 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1418.720699] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1418.724119] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1418.729360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1418.731794] Lustre: Skipped 3 previous similar messages [ 1421.908524] Lustre: server umount lustre-MDT0000 complete [ 1423.077214] LustreError: 64526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781683659 with bad export cookie 1668758758276149458 [ 1423.080153] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1423.082604] LustreError: 64526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1423.184589] Lustre: server umount lustre-MDT0001 complete [ 1425.209405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1426.199069] Lustre: DEBUG MARKER: files compared the same [ 1437.089356] Lustre: server umount lustre-OST0000 complete [ 1442.292533] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 1443.237199] Key type lgssc unregistered [ 1443.357326] LNet: 74245:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1443.359347] LNetError: 74245:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1443.365257] LNet: Removed LNI 192.168.204.144@tcp [ 1443.620497] Key type .llcrypt unregistered [ 1443.621565] Key type ._llcrypt unregistered [ 1451.977114] Key type ._llcrypt registered [ 1451.978123] Key type .llcrypt registered [ 1452.012811] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 1452.325500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1452.380075] alg: No test for adler32 (adler32-zlib) [ 1453.222322] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 1453.298729] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 1454.879089] Key type lgssc registered [ 1455.231749] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1457.849902] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1457.858273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1459.009165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1460.368063] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug +malloc all [ 1463.116827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1463.215372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1464.461719] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug +malloc all [ 1466.713307] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1468.465550] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1470.718730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1470.809405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1472.738382] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug +malloc all [ 1474.988559] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1484.130207] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1489.376464] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1489.376526] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1489.376731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1489.379176] LustreError: Skipped 1 previous similar message [ 1489.386995] Lustre: Skipped 1 previous similar message [ 1489.995725] Lustre: server umount lustre-OST0000 complete [ 1493.324434] Lustre: server umount lustre-MDT0000 complete [ 1493.983505] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1493.986320] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1493.990492] LustreError: 75639:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1493.997190] LustreError: 75639:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1494.477564] LustreError: 75618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781683731 with bad export cookie 13158112949548400851 [ 1494.479319] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1494.483697] LustreError: 75618:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1494.585565] Lustre: server umount lustre-MDT0001 complete [ 1497.941346] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 1498.899511] Key type lgssc unregistered [ 1499.016445] LNet: 79182:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1499.019483] LNetError: 79182:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1499.028311] LNet: Removed LNI 192.168.204.144@tcp [ 1499.290122] Key type .llcrypt unregistered [ 1499.291515] Key type ._llcrypt unregistered [ 1507.608119] Key type ._llcrypt registered [ 1507.609119] Key type .llcrypt registered [ 1507.641441] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 1508.047818] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1508.060379] alg: No test for adler32 (adler32-zlib) [ 1508.915906] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 1508.997380] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 1510.575556] Key type lgssc registered [ 1510.922567] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1513.370362] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1513.378220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1528.799273] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1533.919227] LustreError: 80548:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.144@tcp: failed processing log, type 4: rc = -110 [ 1560.543163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1562.306782] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1564.537228] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1567.019600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1567.057042] Lustre: *** cfs_fail_loc=706, val=0*** [ 1567.186207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1567.214560] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1568.460626] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1571.287222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1571.409695] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1572.767313] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1574.955329] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1576.601989] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1577.439853] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1577.442541] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1577.446818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1581.028786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1581.028972] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1583.380305] Lustre: server umount lustre-OST0000 complete [ 1586.145305] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1586.146254] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1586.149465] Lustre: Skipped 1 previous similar message [ 1586.151030] Lustre: Skipped 1 previous similar message [ 1590.738797] Lustre: server umount lustre-MDT0000 complete [ 1591.264056] LustreError: 83411:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1591.270152] LustreError: 83411:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1591.910978] LustreError: 81659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781683828 with bad export cookie 13171653990534795764 [ 1591.913939] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1591.915351] LustreError: 81659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1592.012875] Lustre: server umount lustre-MDT0001 complete [ 1595.306777] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 1596.268185] Key type lgssc unregistered [ 1596.398290] LNet: 84160:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1596.400816] LNetError: 84160:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1596.408281] LNet: Removed LNI 192.168.204.144@tcp [ 1596.698926] Key type .llcrypt unregistered [ 1596.699982] Key type ._llcrypt unregistered [ 1605.457871] Key type ._llcrypt registered [ 1605.459194] Key type .llcrypt registered [ 1605.494282] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 1605.919185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1605.926525] alg: No test for adler32 (adler32-zlib) [ 1606.767044] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 1606.836431] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 1608.415099] Key type lgssc registered [ 1608.740785] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1611.040449] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1611.046716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1613.252855] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1615.629529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1615.780094] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1617.021053] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1619.652859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1619.746319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1621.520856] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1624.025703] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1624.029790] Lustre: MGS: Client deaa4783-fccc-4ad0-8afd-8fa8b08fff5f (at 0@lo) reconnecting [ 1624.104896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1624.129306] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1625.310623] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1634.272213] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1634.274821] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1634.279101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1636.322726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1636.322881] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1636.326406] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1639.116039] Lustre: server umount lustre-OST0000 complete [ 1641.440466] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1641.440473] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1641.441113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1641.451807] Lustre: Skipped 2 previous similar messages [ 1646.403712] Lustre: server umount lustre-MDT0000 complete [ 1646.560105] LustreError: 86242:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1646.567285] LustreError: 86242:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 1647.663774] Lustre: server umount lustre-MDT0001 complete [ 1648.826655] Lustre: server umount MGS complete [ 1652.388099] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 04:11:28 (1781683888) [ 1661.208928] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 1664.334959] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 1675.395980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1676.672705] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1680.373293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1680.452085] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1680.469417] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1680.504046] Lustre: lustre-MDT0001: new disk, initializing [ 1680.526799] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1680.536867] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1680.539895] Lustre: 93398:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1680.543482] LustreError: 93398:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1681.777224] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1685.738788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1685.829746] Lustre: lustre-OST0000: new disk, initializing [ 1685.831571] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1685.834825] Lustre: 94166:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1685.856358] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1687.775733] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1691.615985] Lustre: 94785:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000c5a04240 x1868230917391104/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1781683944 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1691.649440] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1694.042661] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1694.047767] Lustre: MGS: Client 59709415-10a5-428b-bde7-91a2d1b4f648 (at 0@lo) reconnecting [ 1694.064232] Lustre: 95210:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1694.073742] LustreError: 93382:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1694.076777] LustreError: 93382:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 1694.082422] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1694.118084] Lustre: lustre-MDT0000: new disk, initializing [ 1694.161728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1694.171709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1695.200786] Lustre: 95236:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000f868e919 x1868230917396608/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1781683948 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1695.263500] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1695.743331] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1700.258111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1700.261634] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1700.328878] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 1720.799665] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1720.801849] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1720.805486] Lustre: Skipped 1 previous similar message [ 1720.806962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1725.705859] Lustre: server umount lustre-OST0000 complete [ 1728.479848] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1728.482762] LustreError: Skipped 1 previous similar message [ 1728.484141] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1728.487449] Lustre: Skipped 1 previous similar message [ 1728.489305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1728.490863] Lustre: Skipped 2 previous similar messages [ 1736.672137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1736.674221] Lustre: Skipped 3 previous similar messages [ 1741.279159] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1741.323811] Lustre: server umount lustre-MDT0000 complete [ 1741.792464] LustreError: 93368:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1741.797996] LustreError: 93368:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1748.770761] Lustre: server umount lustre-MDT0001 complete [ 1749.980230] Lustre: server umount MGS complete [ 1753.513736] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 04:13:09 (1781683989) [ 1759.008555] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 1760.023612] Key type lgssc unregistered [ 1760.152396] LNet: 97730:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1760.155882] LNetError: 97730:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1760.165279] LNet: Removed LNI 192.168.204.144@tcp [ 1760.462646] Key type .llcrypt unregistered [ 1760.463527] Key type ._llcrypt unregistered [ 1767.166680] Key type ._llcrypt registered [ 1767.167906] Key type .llcrypt registered [ 1767.209790] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 1767.627709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1767.659578] alg: No test for adler32 (adler32-zlib) [ 1768.511585] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 1768.583595] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 1770.167085] Key type lgssc registered [ 1770.466040] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1772.766082] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1772.772677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1773.912797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1775.122162] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1777.689048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1777.785843] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1778.927470] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1781.005938] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1782.559905] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1784.750838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1784.856393] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1786.811814] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1789.269843] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1799.208359] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 1803.232098] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1803.234518] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1803.240117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1804.258111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1804.260680] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1804.263567] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1805.103097] Lustre: server umount lustre-OST0000 complete [ 1808.864272] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1808.868357] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1808.875800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1808.882238] Lustre: Skipped 1 previous similar message [ 1813.209693] Lustre: server umount lustre-MDT0000 complete [ 1814.406185] LustreError: 98911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781684051 with bad export cookie 17816024445760625278 [ 1814.409242] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1814.410015] LustreError: 98911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1814.511421] Lustre: server umount lustre-MDT0001 complete [ 1818.016552] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 1819.027072] Key type lgssc unregistered [ 1819.144352] LNet: 102476:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1819.146620] LNetError: 102476:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1819.155289] LNet: Removed LNI 192.168.204.144@tcp [ 1819.455692] Key type .llcrypt unregistered [ 1819.457404] Key type ._llcrypt unregistered [ 1825.875976] Key type ._llcrypt registered [ 1825.876969] Key type .llcrypt registered [ 1825.909561] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 1826.224802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1826.231499] alg: No test for adler32 (adler32-zlib) [ 1827.076655] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 1827.162134] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 1828.743133] Key type lgssc registered [ 1829.107619] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1830.954027] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1830.955877] LustreError: 103537:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 1830.956618] Lustre: Skipped 1 previous similar message [ 1830.956625] LustreError: 103536:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 1830.962393] LustreError: 103537:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4996 [ 1830.964351] LustreError: 103537:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1830.967218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1830.969698] LustreError: 103537:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1832.113896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1834.555128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1834.646543] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1835.971935] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1837.496539] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing lsmod [ 1838.428745] LustreError: 104740:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 1838.428923] LustreError: 104741:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 1838.434654] LustreError: 104740:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4997 [ 1838.436880] LustreError: 104740:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1838.439514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1838.442111] LustreError: 104740:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1838.548942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1839.407110] LustreError: 103603:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1839.410076] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1839.410827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1839.412456] LustreError: 103603:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1839.415988] Lustre: Skipped 1 previous similar message [ 1839.418434] LustreError: 103603:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -5 [ 1844.576763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1844.578995] Lustre: Skipped 4 previous similar messages [ 1845.593912] Lustre: server umount lustre-MDT0000 complete [ 1846.845415] LustreError: 103551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781684083 with bad export cookie 12608534133273486781 [ 1846.849055] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1846.849722] LustreError: 103551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1848.671361] LustreError: 104004:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000f1c7475d x1868231148058752/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1848.729792] Lustre: server umount lustre-MDT0001 complete [ 1860.249945] Lustre: server umount lustre-OST0000 complete [ 1863.057123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1863.208194] LustreError: 105807:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1863.215867] LustreError: 105807:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1863.229415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1864.619538] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1868.134401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1868.293072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1869.918444] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1872.704561] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1874.654141] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1877.204199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1877.377363] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1879.682178] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1882.146509] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1889.702920] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 1894.882590] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1894.882944] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1894.884047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1894.887852] Lustre: Skipped 2 previous similar messages [ 1896.015417] Lustre: server umount lustre-OST0000 complete [ 1898.976109] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1898.979417] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1898.984731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1898.986705] Lustre: Skipped 1 previous similar message [ 1899.418726] Lustre: server umount lustre-MDT0000 complete [ 1900.000302] LustreError: 105807:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1900.006785] LustreError: 105807:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1900.803320] LustreError: 105788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781684137 with bad export cookie 12608534133273487432 [ 1900.806729] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1900.810461] LustreError: 105788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1900.948726] Lustre: server umount lustre-MDT0001 complete [ 1904.986759] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 1906.127467] Key type lgssc unregistered [ 1906.256313] LNet: 109354:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1906.259289] LNetError: 109354:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1906.268300] LNet: Removed LNI 192.168.204.144@tcp [ 1906.587872] Key type .llcrypt unregistered [ 1906.589122] Key type ._llcrypt unregistered [ 1915.310629] Key type ._llcrypt registered [ 1915.311550] Key type .llcrypt registered [ 1915.347456] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 1915.659991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1915.707694] alg: No test for adler32 (adler32-zlib) [ 1916.555048] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 1916.624639] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 1918.199223] Key type lgssc registered [ 1918.530675] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1920.846988] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1920.853202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1922.000287] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1923.268977] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1926.055756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1926.173713] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1927.482416] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1929.787861] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1931.412457] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1933.492911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1933.597776] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1935.442438] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1937.730418] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1947.945603] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 1950.341943] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1951.064058] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1953.249943] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1953.250295] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1953.251225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1953.254411] Lustre: Skipped 1 previous similar message [ 1954.064934] Lustre: server umount lustre-OST0000 complete [ 1956.832417] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1956.837755] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1956.842576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1956.851624] Lustre: Skipped 1 previous similar message [ 1957.464384] Lustre: server umount lustre-MDT0000 complete [ 1958.368487] LustreError: 111475:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1958.368550] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1958.375787] LustreError: 111475:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1958.739196] LustreError: 113298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781684195 with bad export cookie 4389075306905844298 [ 1958.741292] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1958.744117] LustreError: 113298:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1958.860744] Lustre: server umount lustre-MDT0001 complete [ 1962.565087] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 1963.543669] Key type lgssc unregistered [ 1963.683374] LNet: 114398:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1963.685634] LNetError: 114398:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1963.695809] LNet: Removed LNI 192.168.204.144@tcp [ 1963.987688] Key type .llcrypt unregistered [ 1963.989381] Key type ._llcrypt unregistered [ 1971.086948] Key type ._llcrypt registered [ 1971.088091] Key type .llcrypt registered [ 1971.125875] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 1971.421102] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1971.449158] alg: No test for adler32 (adler32-zlib) [ 1972.312928] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 1972.400861] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 1973.983139] Key type lgssc registered [ 1974.319100] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1976.770213] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1976.777762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1977.926890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1979.188385] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1981.800953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1981.909905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1983.210339] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1985.534466] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1987.299245] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1989.366055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1989.475155] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1991.300287] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 1993.503565] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2002.789117] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 2007.007975] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2007.010796] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2007.015175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2008.033202] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2008.036161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2008.036829] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2012.880928] Lustre: server umount lustre-OST0000 complete [ 2017.759688] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2017.761819] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2017.765334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2017.767078] Lustre: Skipped 1 previous similar message [ 2020.242803] Lustre: server umount lustre-MDT0000 complete [ 2021.480173] LustreError: 115579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781684258 with bad export cookie 12191719384586532921 [ 2021.483222] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2021.484484] LustreError: 115579:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2021.597437] Lustre: server umount lustre-MDT0001 complete [ 2025.033688] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 2025.989892] Key type lgssc unregistered [ 2026.115616] LNet: 119148:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2026.119574] LNetError: 119148:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2026.129285] LNet: Removed LNI 192.168.204.144@tcp [ 2026.400930] Key type .llcrypt unregistered [ 2026.402039] Key type ._llcrypt unregistered [ 2035.068641] Key type ._llcrypt registered [ 2035.070203] Key type .llcrypt registered [ 2035.106992] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 2035.412310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2035.441372] alg: No test for adler32 (adler32-zlib) [ 2036.296395] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 2036.373135] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 2037.951122] Key type lgssc registered [ 2038.274434] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2040.723664] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2040.729590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2041.875789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2043.145123] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2045.787502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2045.911916] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2047.170643] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2049.384179] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2051.122850] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2053.278522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2053.381872] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2055.185958] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2057.441734] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2066.725321] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 2070.168289] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2071.342645] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2071.345400] Lustre: Skipped 2 previous similar messages [ 2072.683986] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2072.686426] Lustre: Skipped 2 previous similar messages [ 2076.666090] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2076.677466] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2081.947713] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.44@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2081.951843] Lustre: Skipped 2 previous similar messages [ 2084.024913] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.44@tcp 192.168.0.[2,10]@tcp [ 2090.089497] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.44/24@tcp in log lustre-MDT0000 [ 2090.092907] Lustre: Skipped 2 previous similar messages [ 2090.743490] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.44/24@tcp [ 2090.745747] Lustre: Skipped 1 previous similar message [ 2097.632687] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2097.632995] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2097.634096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2097.643638] Lustre: Skipped 1 previous similar message [ 2102.752165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2102.755028] Lustre: Skipped 2 previous similar messages [ 2103.187087] Lustre: server umount lustre-OST0000 complete [ 2107.359978] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2107.363857] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2107.368719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2110.615800] Lustre: server umount lustre-MDT0000 complete [ 2111.825212] LustreError: 120520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781684348 with bad export cookie 5226678770761382365 [ 2111.827914] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2111.830541] LustreError: 120520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2111.941263] Lustre: server umount lustre-MDT0001 complete [ 2115.445088] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 2116.463300] Key type lgssc unregistered [ 2116.587598] LNet: 125396:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2116.591807] LNetError: 125396:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2116.599297] LNet: Removed LNI 192.168.204.144@tcp [ 2116.882855] Key type .llcrypt unregistered [ 2116.884029] Key type ._llcrypt unregistered [ 2127.062246] Key type ._llcrypt registered [ 2127.063208] Key type .llcrypt registered [ 2127.098776] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 2127.376886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2127.392392] alg: No test for adler32 (adler32-zlib) [ 2128.249844] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 2128.334950] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 2129.911213] Key type lgssc registered [ 2130.267271] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2132.899861] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2132.908156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2134.062143] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2134.067256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2134.071319] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.44/24@tcp [ 2135.455135] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2138.371575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2138.495820] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2138.507443] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.44/24@tcp [ 2138.523678] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2139.930727] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2142.546571] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2144.436950] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2146.804081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2146.929620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2148.948336] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2151.419791] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2158.187733] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 2168.802388] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2168.803560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2168.806925] Lustre: Skipped 1 previous similar message [ 2168.811602] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2171.020356] Lustre: server umount lustre-OST0000 complete [ 2173.920550] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2173.921238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2173.924556] Lustre: Skipped 1 previous similar message [ 2173.926590] Lustre: Skipped 1 previous similar message [ 2178.384948] Lustre: server umount lustre-MDT0000 complete [ 2179.039917] LustreError: 126932:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2179.045363] LustreError: 126932:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2179.552151] LustreError: 126915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781684416 with bad export cookie 15079922604405816389 [ 2179.555019] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2179.555310] LustreError: 126915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2179.670517] Lustre: server umount lustre-MDT0001 complete [ 2183.077301] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 2184.027909] Key type lgssc unregistered [ 2184.162347] LNet: 130536:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2184.164981] LNetError: 130536:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2184.176303] LNet: Removed LNI 192.168.204.144@tcp [ 2184.462325] Key type .llcrypt unregistered [ 2184.463850] Key type ._llcrypt unregistered [ 2192.934600] Key type ._llcrypt registered [ 2192.936089] Key type .llcrypt registered [ 2192.973989] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 2199.187654] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 2199.510423] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2199.554716] alg: No test for adler32 (adler32-zlib) [ 2200.411113] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 2200.498934] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 2202.079143] Key type lgssc registered [ 2202.451568] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2213.780035] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2213.785710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2214.875490] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2214.887175] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2214.926092] Lustre: lustre-MDT0000: new disk, initializing [ 2214.952530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2214.958707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2216.320261] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2220.492772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2220.534183] Lustre: 133906:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2220.560252] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2220.563277] Lustre: Skipped 1 previous similar message [ 2220.602514] Lustre: lustre-MDT0001: new disk, initializing [ 2220.634669] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2220.646931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2220.650367] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2221.964587] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2225.665741] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2227.498161] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2229.763813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2229.858361] Lustre: lustre-OST0000: new disk, initializing [ 2229.860142] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2229.862463] Lustre: 135882:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2229.889396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2231.781795] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2235.547838] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2236.954600] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2239.992370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2239.995252] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2240.037541] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2240.096372] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 2241.420887] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2241.487749] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2245.089444] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2245.089801] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2245.091885] LustreError: Skipped 1 previous similar message [ 2245.096136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2245.099183] Lustre: Skipped 1 previous similar message [ 2248.210632] Lustre: server umount lustre-OST0000 complete [ 2250.208546] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2250.209236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2250.212701] Lustre: Skipped 2 previous similar messages [ 2250.216334] Lustre: Skipped 1 previous similar message [ 2255.328035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2255.331488] Lustre: Skipped 1 previous similar message [ 2255.525954] Lustre: server umount lustre-MDT0000 complete [ 2256.810188] LustreError: 133899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781684493 with bad export cookie 13311166343201017630 [ 2256.813168] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2256.814493] LustreError: 133899:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2256.936931] Lustre: server umount lustre-MDT0001 complete [ 2259.789246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2259.954483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2261.267477] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2263.969964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2264.079212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2265.298128] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2267.491578] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2269.185803] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2271.238459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2273.150861] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2275.479597] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2276.950512] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2286.172760] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 2287.544727] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2287.614240] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2290.938909] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2291.002388] Lustre: lustre-OST0001: new disk, initializing [ 2291.004221] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2291.006734] Lustre: 141548:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2291.034701] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2291.037582] Lustre: Skipped 1 previous similar message [ 2292.886684] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2296.197775] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2300.927893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2300.932129] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2300.958874] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2301.355670] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 2302.736725] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2302.813255] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2304.649750] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d5002c62000.ost_server_uuid 50 [ 2305.156362] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8d5002c62000.ost_server_uuid in FULL state after 0 sec [ 2311.139074] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2311.140538] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2311.144048] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2311.146761] LustreError: Skipped 1 previous similar message [ 2311.149296] Lustre: Skipped 1 previous similar message [ 2313.213828] Lustre: server umount lustre-OST0001 complete [ 2314.719806] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2314.723112] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2314.726887] Lustre: Skipped 1 previous similar message [ 2316.257977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2316.258697] LustreError: 139937:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2316.260396] Lustre: Skipped 2 previous similar messages [ 2316.266250] LustreError: 139937:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2320.522342] Lustre: server umount lustre-OST0000 complete [ 2325.471669] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2325.473958] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2325.477937] Lustre: Skipped 1 previous similar message [ 2325.479907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2336.223058] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2336.280673] Lustre: server umount lustre-MDT0000 complete [ 2336.735923] LustreError: 138136:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2336.740318] LustreError: 138136:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2337.412269] LustreError: 138098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781684574 with bad export cookie 13311166343201018505 [ 2337.415113] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2337.415682] LustreError: 138098:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2337.518752] Lustre: server umount lustre-MDT0001 complete [ 2343.322440] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 2344.278406] Key type lgssc unregistered [ 2344.400436] LNet: 144588:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2344.402956] LNetError: 144588:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2344.410245] LNet: Removed LNI 192.168.204.144@tcp [ 2344.694562] Key type .llcrypt unregistered [ 2344.695632] Key type ._llcrypt unregistered [ 2361.233611] Key type ._llcrypt registered [ 2361.234849] Key type .llcrypt registered [ 2361.269836] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 2367.447923] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 2367.906109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2367.916876] alg: No test for adler32 (adler32-zlib) [ 2368.769152] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 2368.853500] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 2370.439114] Key type lgssc registered [ 2370.796473] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2381.987795] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2381.992359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2383.075204] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2383.086203] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2383.123153] Lustre: lustre-MDT0000: new disk, initializing [ 2383.147237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2383.154483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2384.488739] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2388.178888] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2390.371960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2390.477393] Lustre: lustre-OST0000: new disk, initializing [ 2390.479103] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2390.481982] Lustre: Skipped 1 previous similar message [ 2390.484485] Lustre: 150310:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2390.508697] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2392.393690] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2395.973505] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2397.243579] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2399.741792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2399.745684] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2399.753494] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 2400.362083] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 2403.933029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2403.979279] Lustre: 149315:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2404.009802] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2404.045300] Lustre: lustre-MDT0001: new disk, initializing [ 2404.086448] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2404.098819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 2404.103056] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 2405.464736] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2407.943327] Lustre: DEBUG MARKER: MDT2 addition finished [ 2410.108531] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2410.177280] Lustre: lustre-OST0001: new disk, initializing [ 2410.179824] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2410.183587] Lustre: 152537:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2410.213037] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2412.076587] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2412.281421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2412.285959] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2412.325646] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2414.468733] Lustre: DEBUG MARKER: OST2 addition finished [ 2417.638425] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2417.638677] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2417.640513] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2417.641439] LustreError: Skipped 1 previous similar message [ 2417.649766] Lustre: Skipped 1 previous similar message [ 2422.142815] Lustre: server umount lustre-OST0001 complete [ 2424.800078] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2424.802792] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2424.807960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2424.809702] Lustre: Skipped 1 previous similar message [ 2427.872149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2427.874098] LustreError: 150294:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2427.874370] Lustre: Skipped 2 previous similar messages [ 2427.880054] LustreError: 150294:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2429.516747] Lustre: server umount lustre-OST0000 complete [ 2432.992851] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2432.993833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2432.997089] Lustre: Skipped 2 previous similar messages [ 2433.002012] Lustre: Skipped 1 previous similar message [ 2436.754932] Lustre: server umount lustre-MDT0000 complete [ 2437.905495] LustreError: 151707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781684674 with bad export cookie 17614295363143098155 [ 2437.908991] LustreError: 151707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2437.909740] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2438.026759] Lustre: server umount lustre-MDT0001 complete [ 2444.018936] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 2444.939505] Key type lgssc unregistered [ 2445.060368] LNet: 154903:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2445.062723] LNetError: 154903:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2445.071309] LNet: Removed LNI 192.168.204.144@tcp [ 2445.337101] Key type .llcrypt unregistered [ 2445.338377] Key type ._llcrypt unregistered [ 2457.272517] Key type ._llcrypt registered [ 2457.273596] Key type .llcrypt registered [ 2457.306465] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 2462.824955] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 2463.162499] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2463.194948] alg: No test for adler32 (adler32-zlib) [ 2464.048481] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 2464.126467] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 2465.703148] Key type lgssc registered [ 2466.072338] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2477.008930] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2477.015530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2478.102761] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2478.115303] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2478.149986] Lustre: lustre-MDT0000: new disk, initializing [ 2478.174310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2478.181868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2479.507100] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2483.482915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2483.528302] Lustre: 159028:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2483.553717] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2483.555838] Lustre: Skipped 1 previous similar message [ 2483.589709] Lustre: lustre-MDT0001: new disk, initializing [ 2483.621277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2483.633482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2483.637253] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2484.841648] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2488.337139] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2490.031392] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2492.147040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2492.254376] Lustre: lustre-OST0000: new disk, initializing [ 2492.256109] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2492.259320] Lustre: 160994:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2492.293967] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2494.129822] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2495.992306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2495.995417] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2496.029109] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2497.622833] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2499.495307] Lustre: Failing over lustre-OST0000 [ 2499.560771] Lustre: server umount lustre-OST0000 complete [ 2512.182165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2512.283717] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2512.288562] Lustre: Skipped 1 previous similar message [ 2512.293482] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2512.299458] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2512.977118] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2514.104083] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2515.270336] Lustre: Failing over lustre-MDT0000 [ 2515.360668] Lustre: server umount lustre-MDT0000 complete [ 2516.319213] Lustre: 157256:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781684737/real 1781684737] req@0000000040d7d1ac x1868231816014464/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781684753 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2516.327050] Lustre: 157256:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2517.472544] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2517.475988] Lustre: Skipped 2 previous similar messages [ 2522.591180] Lustre: 157257:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781684743/real 1781684743] req@000000009df7f7c4 x1868231816015616/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781684759 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2522.592059] LustreError: 159038:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2522.604562] LustreError: 159038:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2523.401408] LustreError: 159033:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2527.711172] Lustre: 157255:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781684748/real 1781684748] req@00000000999e0b59 x1868231816016768/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781684764 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2527.714275] LustreError: 161002:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2527.728118] LustreError: 161002:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2528.152062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2528.186871] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2528.191289] Lustre: 159030:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.204.144@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2528.260453] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2528.277950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2528.301492] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2528.521463] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2529.529782] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2533.349844] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2533.356662] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2533.360339] Lustre: Skipped 1 previous similar message [ 2533.376085] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2538.467368] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2538.467986] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2538.474536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2538.476082] Lustre: Skipped 1 previous similar message [ 2540.426321] Lustre: server umount lustre-OST0000 complete [ 2543.584614] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2543.585121] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2543.585313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2543.587783] Lustre: Skipped 2 previous similar messages [ 2543.590701] LustreError: Skipped 1 previous similar message [ 2547.734057] Lustre: server umount lustre-MDT0000 complete [ 2548.704092] LustreError: 159039:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2548.711055] LustreError: 159039:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2548.958169] LustreError: 160993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781684785 with bad export cookie 4598329103045664295 [ 2548.961198] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2548.962040] LustreError: 160993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2549.069449] Lustre: server umount lustre-MDT0001 complete [ 2552.516800] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 2553.574879] Key type lgssc unregistered [ 2553.695423] LNet: 164637:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2553.698391] LNetError: 164637:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2553.706416] LNet: Removed LNI 192.168.204.144@tcp [ 2554.017745] Key type .llcrypt unregistered [ 2554.019396] Key type ._llcrypt unregistered [ 2574.038813] Key type ._llcrypt registered [ 2574.040173] Key type .llcrypt registered [ 2574.077584] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 2574.491979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2574.511891] alg: No test for adler32 (adler32-zlib) [ 2575.357288] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 2575.433582] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 2577.007113] Key type lgssc registered [ 2577.343664] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2579.800700] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2579.808599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2580.956551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2582.259795] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2585.663161] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2587.583674] LustreError: 166183:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2587.587201] LustreError: 166183:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2587.589887] LustreError: 166183:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 2587.695829] Lustre: server umount lustre-MDT0000 complete [ 2592.066529] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 2594.975268] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 2605.574289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2605.662520] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2605.672990] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2605.709519] Lustre: lustre-MDT0000: new disk, initializing [ 2605.733084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2605.740173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2607.067315] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2611.072625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2611.113304] Lustre: 169480:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2611.136927] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2611.138710] Lustre: Skipped 1 previous similar message [ 2611.173512] Lustre: lustre-MDT0001: new disk, initializing [ 2611.204331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2611.215798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2611.219184] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2612.508536] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2616.193750] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2617.970815] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2620.321912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2620.430189] Lustre: lustre-OST0000: new disk, initializing [ 2620.431883] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2620.434430] Lustre: 171443:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2620.462129] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2621.303831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2621.307784] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2621.340864] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2622.513056] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2626.306705] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2628.151867] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2628.687119] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 04:27:44 (1781684864) [ 2693.087605] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2693.087702] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2693.090840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2693.094235] Lustre: Skipped 2 previous similar messages [ 2693.097203] Lustre: Skipped 1 previous similar message [ 2698.207483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2698.210189] Lustre: Skipped 2 previous similar messages [ 2703.327402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2703.329558] Lustre: Skipped 2 previous similar messages [ 2706.399083] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2706.467105] Lustre: server umount lustre-MDT0000 complete [ 2707.641498] LustreError: 169472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781684944 with bad export cookie 12024954339042847455 [ 2707.642635] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2707.647128] LustreError: 169472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2708.447480] LustreError: 169485:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2708.447603] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2708.452036] LustreError: 169485:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2708.458776] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2712.479353] LustreError: 169487:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2712.479394] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2713.820378] Lustre: server umount lustre-MDT0001 complete [ 2721.093899] Lustre: server umount lustre-OST0000 complete [ 2723.914563] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 2726.674515] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 2737.825053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2737.917275] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2737.928984] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2737.972560] Lustre: lustre-MDT0000: new disk, initializing [ 2737.999731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2738.007380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2739.484319] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2743.681844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2743.722868] Lustre: 176071:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2743.753292] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2743.753406] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2743.755233] Lustre: Skipped 1 previous similar message [ 2743.794328] Lustre: lustre-MDT0001: new disk, initializing [ 2743.839752] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2743.844563] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2745.195790] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2748.702973] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2750.424967] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2752.536228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2752.626446] Lustre: lustre-OST0000: new disk, initializing [ 2752.629264] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2752.632230] Lustre: 178041:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2752.659959] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2752.662240] Lustre: Skipped 1 previous similar message [ 2753.782731] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2753.785806] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2753.818655] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2754.409843] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2757.930667] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2759.195047] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2759.251095] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2760.438697] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2760.503259] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2763.161318] Lustre: server umount lustre-OST0000 complete [ 2769.377685] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2769.378277] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2769.378817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2769.386115] Lustre: Skipped 1 previous similar message [ 2774.522388] Lustre: server umount lustre-MDT0000 complete [ 2775.698382] LustreError: 176064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781685012 with bad export cookie 12024954339043006271 [ 2775.700552] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2775.704565] LustreError: 176064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2785.294441] Lustre: server umount lustre-MDT0001 complete [ 2788.060909] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 04:30:24 (1781685024) [ 2793.382726] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 2796.568075] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 2808.337849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2808.422859] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2808.436829] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2808.474832] Lustre: lustre-MDT0000: new disk, initializing [ 2808.505413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2808.514180] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2809.834061] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2813.790405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2813.831883] Lustre: 182788:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2813.918268] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2815.151059] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2818.790341] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2820.535793] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2822.740925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2822.841288] Lustre: 184762:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2824.780918] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2825.847954] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2825.887721] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2828.582579] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2831.329135] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2831.329563] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2831.332457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2831.336050] Lustre: Skipped 1 previous similar message [ 2831.339677] Lustre: Skipped 4 previous similar messages [ 2836.568697] Lustre: server umount lustre-OST0000 complete [ 2839.519983] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2839.522348] LustreError: Skipped 1 previous similar message [ 2839.523546] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2852.319100] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2852.378107] Lustre: server umount lustre-MDT0000 complete [ 2853.499142] LustreError: 182778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781685090 with bad export cookie 12024954339043007125 [ 2853.502100] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2853.504680] LustreError: 182778:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2853.608749] Lustre: server umount lustre-MDT0001 complete [ 2856.235104] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 04:31:32 (1781685092) [ 2861.230145] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 2864.219034] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 2875.155214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2875.249966] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2875.263268] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2875.265155] Lustre: Skipped 3 previous similar messages [ 2875.300661] Lustre: lustre-MDT0000: new disk, initializing [ 2875.302856] Lustre: Skipped 2 previous similar messages [ 2875.328031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2875.329890] Lustre: Skipped 2 previous similar messages [ 2875.336280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2875.339150] Lustre: Skipped 2 previous similar messages [ 2876.706462] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2880.680758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2880.719174] Lustre: 189073:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2880.809532] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2882.033927] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2885.528871] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2887.198633] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2889.247902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2889.346883] Lustre: 191050:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2891.364209] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2891.553757] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2895.002770] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2896.864292] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2896.865348] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2896.866091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2896.866095] Lustre: Skipped 9 previous similar messages [ 2896.869032] LustreError: Skipped 1 previous similar message [ 2896.874341] Lustre: Skipped 2 previous similar messages [ 2902.733229] Lustre: server umount lustre-OST0000 complete [ 2906.591754] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2918.367103] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2918.420368] Lustre: server umount lustre-MDT0000 complete [ 2919.553283] LustreError: 189065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781685156 with bad export cookie 12024954339043008196 [ 2919.556836] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2919.557721] LustreError: 189065:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2919.668521] Lustre: server umount lustre-MDT0001 complete [ 2923.334790] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 2924.291389] Key type lgssc unregistered [ 2924.407308] LNet: 192905:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2924.409465] LNetError: 192905:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2924.416230] LNet: Removed LNI 192.168.204.144@tcp [ 2924.687115] Key type .llcrypt unregistered [ 2924.688275] Key type ._llcrypt unregistered [ 2933.748125] Key type ._llcrypt registered [ 2933.749340] Key type .llcrypt registered [ 2933.787437] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 2934.187345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2934.228344] alg: No test for adler32 (adler32-zlib) [ 2935.108921] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 2935.188942] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 2936.767133] Key type lgssc registered [ 2937.096265] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2939.661137] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2939.667540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2940.810441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2942.047273] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2944.813453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2944.936665] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2946.189537] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2948.354539] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2950.026475] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2952.105114] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2952.209325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2954.019729] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 2956.264233] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2965.541849] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2979.808057] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2979.812972] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2979.821550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2980.835605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2980.835755] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2980.842172] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2984.466873] Lustre: server umount lustre-OST0000 complete [ 2985.952573] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2985.953040] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2985.953500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2985.953504] Lustre: Skipped 1 previous similar message [ 2985.957852] Lustre: Skipped 1 previous similar message [ 2991.072210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2991.075591] Lustre: Skipped 2 previous similar messages [ 2991.828165] Lustre: server umount lustre-MDT0000 complete [ 2993.098275] LustreError: 197104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781685229 with bad export cookie 6582319668568517583 [ 2993.100539] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2993.103499] LustreError: 197104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2993.224866] Lustre: server umount lustre-MDT0001 complete [ 2997.089749] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 2998.122834] Key type lgssc unregistered [ 2998.260471] LNet: 197854:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2998.263460] LNetError: 197854:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2998.270379] LNet: Removed LNI 192.168.204.144@tcp [ 2998.582561] Key type .llcrypt unregistered [ 2998.584488] Key type ._llcrypt unregistered [ 3007.559504] Key type ._llcrypt registered [ 3007.560703] Key type .llcrypt registered [ 3007.595534] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 3008.029802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3008.040873] alg: No test for adler32 (adler32-zlib) [ 3008.887406] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 3008.977066] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 3010.567179] Key type lgssc registered [ 3010.991279] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3013.704692] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3013.711474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3014.892107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3016.255655] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3019.159459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3019.287939] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3020.694754] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3023.032193] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3024.889318] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3028.263871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3028.442040] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3031.794426] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3036.276648] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3040.751624] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 3044.834759] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3044.838792] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3044.846438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3045.859325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3045.860506] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3045.860654] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3045.863918] Lustre: Skipped 1 previous similar message [ 3050.343301] Lustre: server umount lustre-OST0000 complete [ 3054.534195] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8d500646e800.ost_server_uuid 50 [ 3055.597928] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 3070.432796] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3070.433734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3070.453997] Lustre: Skipped 1 previous similar message [ 3070.463762] Lustre: Skipped 2 previous similar messages [ 3075.357352] Lustre: server umount lustre-MDT0000 complete [ 3076.577181] LustreError: 199246:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3076.591874] LustreError: 199246:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3078.017904] LustreError: 199228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781685314 with bad export cookie 11925737157275941430 [ 3078.025151] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3078.026884] LustreError: 199228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3078.211553] Lustre: server umount lustre-MDT0001 complete [ 3083.331417] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 04:35:19 (1781685319) [ 3087.029128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3087.298177] LustreError: 203155:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3087.340365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3089.530850] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3092.450350] LustreError: 203156:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3093.458454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3093.643089] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3095.440866] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3098.271204] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3100.511587] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3103.309854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3103.430641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3105.615559] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3108.537288] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3111.247730] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3111.333082] Lustre: lustre-OST0001: new disk, initializing [ 3111.335216] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3111.338410] Lustre: 205992:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3112.430613] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3113.940381] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3118.392789] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3120.098147] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3120.102637] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3120.110381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3120.113616] Lustre: Skipped 2 previous similar messages [ 3121.155593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3121.160532] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3121.200871] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3125.726177] Lustre: server umount lustre-OST0000 complete [ 3126.244820] LustreError: 204976:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3126.257677] LustreError: 204976:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3128.952903] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3129.052875] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3130.981435] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3131.082212] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3131.362420] LustreError: 206798:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3131.377091] LustreError: 206798:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3141.601095] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3141.604025] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3141.604605] LustreError: Skipped 1 previous similar message [ 3141.604885] LustreError: 204975:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3141.604905] LustreError: 204975:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3141.609513] Lustre: Skipped 1 previous similar message [ 3141.610316] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3141.626218] Lustre: Skipped 4 previous similar messages [ 3147.539299] Lustre: server umount lustre-OST0001 complete [ 3150.303901] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3150.307632] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3150.315138] Lustre: Skipped 1 previous similar message [ 3150.317815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3150.320752] Lustre: Skipped 2 previous similar messages [ 3155.038567] Lustre: server umount lustre-MDT0000 complete [ 3156.359169] LustreError: 205990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781685393 with bad export cookie 11925737157275942494 [ 3156.360822] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3156.364947] LustreError: 205990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3156.485357] Lustre: server umount lustre-MDT0001 complete [ 3166.528961] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 04:36:42 (1781685402) [ 3169.569207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3169.694727] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3169.714779] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3169.811953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3169.814608] Lustre: Skipped 1 previous similar message [ 3171.486859] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3175.147576] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3175.222150] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3175.245611] Lustre: 209402:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3175.278645] LustreError: 209413:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3175.286764] LustreError: 209413:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3177.069995] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3179.888301] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3182.319743] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3185.285824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3185.383384] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3187.870163] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3188.914389] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3191.170427] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3194.691878] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3194.748660] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3198.032823] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3201.997715] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3204.579368] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3204.579641] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3204.583302] LustreError: Skipped 1 previous similar message [ 3204.591503] Lustre: Skipped 1 previous similar message [ 3204.594659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3204.599114] Lustre: Skipped 3 previous similar messages [ 3209.613220] Lustre: server umount lustre-OST0000 complete [ 3211.233641] LustreError: 211249:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3211.244063] LustreError: 211249:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3221.480973] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3227.432976] Lustre: server umount lustre-OST0001 complete [ 3235.576456] Lustre: server umount lustre-MDT0000 complete [ 3237.501169] LustreError: 209394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781685474 with bad export cookie 11925737157275943852 [ 3237.503488] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3237.510337] LustreError: 209394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3237.675786] Lustre: server umount lustre-MDT0001 complete [ 3249.060912] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 04:38:04 (1781685484) [ 3256.059884] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 3260.315508] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 3275.795603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3275.909534] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3275.930205] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3275.975771] Lustre: lustre-MDT0000: new disk, initializing [ 3276.010670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3276.014814] Lustre: Skipped 3 previous similar messages [ 3276.023334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3277.644345] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3282.300702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3282.347875] Lustre: 217476:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3282.373662] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3282.376126] Lustre: Skipped 1 previous similar message [ 3282.414672] Lustre: lustre-MDT0001: new disk, initializing [ 3282.460573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3282.464915] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3283.939046] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3288.097896] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3290.128468] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3292.671875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3292.776309] Lustre: lustre-OST0000: new disk, initializing [ 3292.778294] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3292.781607] Lustre: 219446:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3294.854416] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3298.836832] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3300.523669] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3301.881154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3301.885249] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3301.916944] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3302.654827] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 3304.260078] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3304.349679] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3306.978807] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3306.979229] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3306.980965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3306.980971] Lustre: Skipped 9 previous similar messages [ 3306.983994] LustreError: Skipped 3 previous similar messages [ 3306.988935] Lustre: Skipped 5 previous similar messages [ 3311.329541] Lustre: server umount lustre-OST0000 complete [ 3319.133763] Lustre: server umount lustre-MDT0000 complete [ 3320.985716] LustreError: 218270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781685557 with bad export cookie 11925737157275945413 [ 3320.990909] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3320.995451] LustreError: 218270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3321.170864] Lustre: server umount lustre-MDT0001 complete [ 3325.047860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3325.281165] LustreError: 221672:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3325.294770] LustreError: 221672:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 3327.151450] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3331.444802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3333.446680] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3336.913464] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3339.632905] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3342.789167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3342.948430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3342.952438] Lustre: Skipped 4 previous similar messages [ 3345.849775] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3349.546357] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3351.611973] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3351.710827] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3353.644894] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3356.812183] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 3 sec [ 3363.802613] Lustre: server umount lustre-OST0000 complete [ 3366.262912] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3366.371470] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3368.199886] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3392.731914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3395.651960] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3398.876727] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3399.617456] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3401.952936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3401.956876] Lustre: Skipped 7 previous similar messages [ 3407.075945] Lustre: server umount lustre-OST0000 complete [ 3412.450478] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3412.458575] Lustre: Skipped 5 previous similar messages [ 3423.199304] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3423.273218] Lustre: server umount lustre-MDT0000 complete [ 3424.945384] LustreError: 226798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781685661 with bad export cookie 11925737157275946281 [ 3424.948346] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3424.952226] LustreError: 226798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3425.107221] Lustre: server umount lustre-MDT0001 complete [ 3428.783494] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 04:41:04 (1781685664) [ 3431.688308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3433.792937] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3437.474606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3439.290887] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3442.369224] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3444.795602] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3448.174694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3450.914899] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3454.076376] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3455.848507] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3456.999156] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3458.976693] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3463.198728] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 4 sec [ 3466.044897] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3466.141128] Lustre: lustre-OST0001: new disk, initializing [ 3466.144440] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3466.149138] Lustre: 231075:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3468.527879] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3472.969320] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3474.448223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3474.453289] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3474.504516] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3474.981350] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3475.076893] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3477.096462] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3477.233593] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3479.521526] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3479.528476] LustreError: Skipped 4 previous similar messages [ 3480.205924] Lustre: server umount lustre-OST0001 complete [ 3483.098614] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3483.201650] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3484.645937] LustreError: 231949:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3484.655940] LustreError: 231949:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3485.227601] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3485.339773] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3509.816635] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3509.934591] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3509.938529] Lustre: Skipped 5 previous similar messages [ 3512.420386] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3515.371444] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3515.378093] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3515.384876] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3515.557107] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3516.308320] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3523.338204] Lustre: server umount lustre-OST0001 complete [ 3535.844948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3535.848017] Lustre: Skipped 12 previous similar messages [ 3550.176818] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3550.291740] Lustre: server umount lustre-OST0000 complete [ 3555.304798] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3555.377124] Lustre: Skipped 6 previous similar messages [ 3560.927231] Lustre: server umount lustre-MDT0000 complete [ 3564.157657] LustreError: 227652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781685800 with bad export cookie 11925737157275947233 [ 3564.159910] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3564.165441] LustreError: 227652:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3570.675263] Lustre: server umount lustre-MDT0001 complete [ 3595.159470] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 04:43:50 (1781685830) [ 3600.791917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3601.041539] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3601.071455] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3605.063477] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3613.717946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3613.892447] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3613.974701] Lustre: 236314:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3618.835963] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3625.100554] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3628.990769] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3633.807934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3633.921959] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3638.508296] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3644.135388] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3651.263022] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3651.303586] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3655.341762] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3660.513617] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3661.770896] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3665.149520] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3665.303895] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3668.213363] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3668.357613] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3672.455765] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d50066a4000.ost_server_uuid 50 [ 3673.656392] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8d50066a4000.ost_server_uuid in FULL state after 0 sec [ 3674.355643] Lustre: Permanently deactivating lustre-OST0001 [ 3674.378225] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3676.997319] Lustre: Permanently reactivating lustre-OST0001 [ 3681.765090] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3681.771833] LustreError: Skipped 3 previous similar messages [ 3682.417791] Lustre: server umount lustre-OST0001 complete [ 3692.131508] Lustre: server umount lustre-OST0000 complete [ 3703.013509] Lustre: server umount lustre-MDT0000 complete [ 3708.794716] LustreError: 236308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781685945 with bad export cookie 11925737157275948507 [ 3708.801114] LustreError: 236308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3708.801901] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3709.138219] Lustre: server umount lustre-MDT0001 complete [ 3732.089494] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 04:46:07 (1781685967) [ 3737.274109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3737.466404] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3737.491490] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3737.502177] Lustre: Skipped 5 previous similar messages [ 3740.547588] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3746.880640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3746.947903] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3746.974096] Lustre: 243001:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3747.029071] LustreError: 243014:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3747.046380] LustreError: 243014:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 29 previous similar messages [ 3750.979262] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3757.197763] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3761.792107] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3766.961661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3767.139533] Lustre: Permanently deactivating lustre-OST0000 [ 3767.152747] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3767.311587] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3767.324553] Lustre: Skipped 6 previous similar messages [ 3772.538756] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3773.486354] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3773.493363] LustreError: 245289:0:(osp_precreate.c:655:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 3773.505482] LustreError: 245289:0:(osp_precreate.c:1359:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 3778.872874] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3784.832898] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3784.875899] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3784.880117] Lustre: Skipped 1 previous similar message [ 3790.599226] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3790.974533] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3797.390575] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3798.957311] Lustre: Permanently reactivating lustre-OST0000 [ 3798.971377] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 3798.976552] Lustre: Skipped 2 previous similar messages [ 3806.236400] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 33 seconds. I think it's dead, and I am evicting it. exp 0000000042f92ca2, cur 1781686043 deadline 1781686040 last 1781686010 [ 3806.238618] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3806.267625] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3806.284502] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3806.289603] Lustre: Skipped 1 previous similar message [ 3806.330652] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3815.664400] Lustre: server umount lustre-OST0001 complete [ 3816.421342] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3816.433203] Lustre: Skipped 8 previous similar messages [ 3821.537039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3821.541077] Lustre: Skipped 18 previous similar messages [ 3833.311353] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3833.474704] Lustre: server umount lustre-OST0000 complete [ 3842.529552] Lustre: server umount lustre-MDT0000 complete [ 3845.191388] LustreError: 246592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781686082 with bad export cookie 11925737157275950159 [ 3845.199457] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3845.203662] LustreError: 246592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3845.477164] Lustre: server umount lustre-MDT0001 complete [ 3852.968470] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 3855.222204] Key type lgssc unregistered [ 3855.447909] LNet: 247884:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3855.455389] LNetError: 247884:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3855.475599] LNet: Removed LNI 192.168.204.144@tcp [ 3856.155599] Key type .llcrypt unregistered [ 3856.157862] Key type ._llcrypt unregistered [ 3883.941740] Key type ._llcrypt registered [ 3883.943979] Key type .llcrypt registered [ 3884.088871] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 3885.011888] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3885.060964] alg: No test for adler32 (adler32-zlib) [ 3886.211387] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 3886.511097] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 3888.218120] Key type lgssc registered [ 3889.183482] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3898.190464] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3898.207986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3899.829998] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3904.409692] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3910.117193] LustreError: 249353:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3910.132796] LustreError: 249353:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3912.210808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3912.318449] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3912.341428] Lustre: 249342:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3912.350923] Lustre: Permanently deactivating lustre-MDT0001 [ 3912.365770] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3912.616334] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3915.919925] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3915.933453] LustreError: 249384:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3915.954479] LustreError: 249384:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3915.961392] LustreError: 249384:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 1, failed: rc = -108 [ 3916.256534] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3923.083492] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3927.966518] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3933.645547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3933.873834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3937.974972] LustreError: 251484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3937.998420] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3939.157216] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3943.400715] LustreError: 251484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3944.440783] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3945.451220] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3949.539329] LustreError: 251195:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3949.563432] LustreError: 251195:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3951.339253] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3951.516253] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3956.727335] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3956.732673] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3957.309845] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 3963.872250] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3967.490434] Lustre: Permanently reactivating lustre-MDT0001 [ 3967.503468] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3967.508299] Lustre: Skipped 2 previous similar messages [ 3970.921620] Lustre: lustre-MDT0001: Client b4499472-9318-4ded-ba93-0361823ae419 (at 192.168.204.44@tcp) reconnecting [ 3976.318672] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3977.724133] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3981.610537] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3981.765885] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3984.689092] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3984.829657] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3986.628914] Lustre: Permanently deactivating lustre-MDT0001 [ 3986.640991] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3986.645557] Lustre: Skipped 2 previous similar messages [ 3996.228465] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 4003.112941] Lustre: server umount lustre-MDT0000 complete [ 4003.811137] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4003.821032] LustreError: 251483:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4003.829021] Lustre: Skipped 2 previous similar messages [ 4003.840882] LustreError: 251483:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4005.849241] LustreError: 249333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781686242 with bad export cookie 15433927211544722130 [ 4005.852316] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4005.861297] LustreError: 249333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4006.106948] Lustre: server umount lustre-MDT0001 complete [ 4008.900268] Lustre: server umount lustre-OST0000 complete [ 4018.078330] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 04:50:53 (1781686253) [ 4027.159521] Lustre: server umount lustre-OST0001 complete [ 4033.772625] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 4042.606399] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 4072.514045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4072.787275] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4072.802073] Lustre: Skipped 2 previous similar messages [ 4072.823138] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4072.883477] Lustre: lustre-MDT0000: new disk, initializing [ 4072.955509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4072.975328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4076.673319] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4085.190526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4085.290332] Lustre: 257619:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4085.352372] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4085.355679] Lustre: Skipped 1 previous similar message [ 4085.417333] Lustre: lustre-MDT0001: new disk, initializing [ 4085.504236] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4085.546190] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4085.555595] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4089.128834] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4097.409696] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4102.357026] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4107.772365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4108.036930] Lustre: lustre-OST0000: new disk, initializing [ 4108.041819] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4108.050764] Lustre: 259585:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4108.124892] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4112.812933] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4114.469094] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4114.477727] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4114.542051] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4120.355140] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4122.280416] LustreError: 257625:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4129.275121] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4129.449137] Lustre: lustre-OST0001: new disk, initializing [ 4129.454763] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4129.469969] Lustre: 260700:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4132.312158] LustreError: 257625:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4132.366493] LustreError: 258583:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4134.984591] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4135.477562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4135.490296] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4135.583128] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4142.383153] LustreError: 258583:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4143.135366] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4144.635813] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4151.079176] Lustre: lustre-OST0001: Not available for connect from 192.168.204.44@tcp (stopping) [ 4151.267619] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4151.270097] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4151.270109] Lustre: Skipped 1 previous similar message [ 4151.304290] LustreError: Skipped 1 previous similar message [ 4152.364550] Lustre: server umount lustre-OST0001 complete [ 4156.399552] LustreError: 259571:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4156.418828] LustreError: 259571:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4170.731301] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4170.732674] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4170.741716] LustreError: Skipped 1 previous similar message [ 4170.752717] Lustre: Skipped 1 previous similar message [ 4170.754441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4170.773752] Lustre: Skipped 3 previous similar messages [ 4174.021351] Lustre: server umount lustre-OST0000 complete [ 4177.893902] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4177.909777] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4177.921062] Lustre: Skipped 1 previous similar message [ 4177.931753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4177.934861] Lustre: Skipped 2 previous similar messages [ 4181.985383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4181.990509] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4183.597533] Lustre: server umount lustre-MDT0000 complete [ 4186.772491] LustreError: 257610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781686423 with bad export cookie 15433927211544724944 [ 4186.773402] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4186.778675] LustreError: 257610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4193.117388] Lustre: server umount lustre-MDT0001 complete [ 4203.163742] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 4205.512232] Key type lgssc unregistered [ 4205.759453] LNet: 262733:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4205.768358] LNetError: 262733:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4205.790883] LNet: Removed LNI 192.168.204.144@tcp [ 4206.589767] Key type .llcrypt unregistered [ 4206.592113] Key type ._llcrypt unregistered [ 4238.084338] Key type ._llcrypt registered [ 4238.087774] Key type .llcrypt registered [ 4238.186841] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 4238.802211] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4238.888540] alg: No test for adler32 (adler32-zlib) [ 4239.878828] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 4240.050596] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 4241.695170] Key type lgssc registered [ 4242.631793] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4248.972177] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4248.988481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4250.219418] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4250.234901] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4250.393519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4253.825400] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4260.111882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4260.196120] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4260.235550] Lustre: 264919:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4260.451327] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4264.255854] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4270.705719] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4275.497549] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4280.986098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4281.209574] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4281.360352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4286.329187] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4291.148048] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 4292.770343] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4342.248361] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4342.264463] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4342.280089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4347.204810] Lustre: server umount lustre-OST0000 complete [ 4350.747633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 4360.921333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4361.201272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4362.215777] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4362.223718] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4362.227803] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4362.244581] Lustre: Skipped 1 previous similar message [ 4366.155464] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4371.793885] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4408.292786] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4408.294116] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4408.305309] Lustre: Skipped 2 previous similar messages [ 4408.311042] LustreError: Skipped 1 previous similar message [ 4408.314855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4408.324703] Lustre: Skipped 2 previous similar messages [ 4413.412528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4414.297368] Lustre: server umount lustre-OST0000 complete [ 4418.530435] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4418.532691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4418.541424] Lustre: Skipped 1 previous similar message [ 4418.544667] Lustre: Skipped 1 previous similar message [ 4423.649419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4423.659961] Lustre: Skipped 1 previous similar message [ 4431.327198] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4431.476969] Lustre: server umount lustre-MDT0000 complete [ 4434.386680] LustreError: 264910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781686671 with bad export cookie 13713076441649590161 [ 4434.389549] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4434.398469] LustreError: 264910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4434.645833] Lustre: server umount lustre-MDT0001 complete [ 4442.846599] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 4444.879755] Key type lgssc unregistered [ 4445.102484] LNet: 270339:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4445.114051] LNetError: 270339:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4445.129820] LNet: Removed LNI 192.168.204.144@tcp [ 4445.736901] Key type .llcrypt unregistered [ 4445.740687] Key type ._llcrypt unregistered [ 4462.343132] Key type ._llcrypt registered [ 4462.345106] Key type .llcrypt registered [ 4462.459564] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 4463.158022] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4463.222295] alg: No test for adler32 (adler32-zlib) [ 4464.235072] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 4464.407988] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 4466.055254] Key type lgssc registered [ 4466.819427] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4472.411420] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4472.426718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4473.882579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4477.132786] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4483.677486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4483.911528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4486.783728] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4491.632550] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4495.537941] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4500.792732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4501.077726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4505.449586] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4510.456355] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4519.533921] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4539.361505] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4539.368584] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4539.384600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4540.385741] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4540.396662] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4540.402435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4540.417205] Lustre: Skipped 1 previous similar message [ 4541.704919] Lustre: server umount lustre-OST0000 complete [ 4545.506303] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4545.511941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4545.512222] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4545.520913] Lustre: Skipped 1 previous similar message [ 4545.534517] Lustre: Skipped 1 previous similar message [ 4550.562996] Lustre: server umount lustre-MDT0000 complete [ 4550.624979] LustreError: 272463:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4550.636557] LustreError: 272463:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4553.201372] LustreError: 271715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781686790 with bad export cookie 10109936223778521146 [ 4553.202757] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4553.209769] LustreError: 271715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4553.450771] Lustre: server umount lustre-MDT0001 complete [ 4560.327665] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 4562.085811] Key type lgssc unregistered [ 4562.279977] LNet: 276064:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4562.286302] LNetError: 276064:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4562.294768] LNet: Removed LNI 192.168.204.144@tcp [ 4562.766309] Key type .llcrypt unregistered [ 4562.768409] Key type ._llcrypt unregistered [ 4575.237491] Key type ._llcrypt registered [ 4575.240083] Key type .llcrypt registered [ 4575.315668] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 4576.028664] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4576.045480] alg: No test for adler32 (adler32-zlib) [ 4577.020257] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 4577.174760] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 4578.823184] Key type lgssc registered [ 4579.674567] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4585.079233] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4585.093397] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4586.474792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4589.233963] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4595.308583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4595.521477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4598.670514] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4604.093166] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4607.700265] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4612.553550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4612.835177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4617.360601] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4623.009890] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4631.336875] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4641.766099] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4641.768708] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4641.778569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4641.793419] Lustre: Skipped 1 previous similar message [ 4646.888811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4646.898236] Lustre: Skipped 2 previous similar messages [ 4647.650535] Lustre: server umount lustre-OST0000 complete [ 4652.004882] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4652.005124] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4652.013795] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4652.032910] Lustre: Skipped 1 previous similar message [ 4656.396480] Lustre: server umount lustre-MDT0000 complete [ 4657.120496] LustreError: 277265:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4657.136525] LustreError: 277265:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4658.781365] LustreError: 277246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781686895 with bad export cookie 8459380178157745869 [ 4658.784475] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4658.794235] LustreError: 277246:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4658.972734] Lustre: server umount lustre-MDT0001 complete [ 4666.127695] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 4667.897509] Key type lgssc unregistered [ 4668.073514] LNet: 280973:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4668.081506] LNetError: 280973:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4668.098146] LNet: Removed LNI 192.168.204.144@tcp [ 4668.697732] Key type .llcrypt unregistered [ 4668.700072] Key type ._llcrypt unregistered [ 4685.480425] Key type ._llcrypt registered [ 4685.483136] Key type .llcrypt registered [ 4685.571262] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 4686.214580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4686.290145] alg: No test for adler32 (adler32-zlib) [ 4687.252256] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 4687.463215] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 4689.095176] Key type lgssc registered [ 4689.787292] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4694.697552] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4694.710334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4695.994822] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4698.410587] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4704.120447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4704.348260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4707.403656] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4712.924330] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4717.127825] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4721.634350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4721.839501] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4725.422715] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4729.954461] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4740.269106] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4755.937068] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4755.938763] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4755.940984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4755.946041] LustreError: Skipped 1 previous similar message [ 4755.969379] Lustre: Skipped 1 previous similar message [ 4761.057564] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4761.062865] Lustre: Skipped 2 previous similar messages [ 4762.068252] Lustre: server umount lustre-OST0000 complete [ 4765.664521] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4765.669746] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4765.677595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4770.894609] Lustre: server umount lustre-MDT0000 complete [ 4771.296604] LustreError: 283173:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4771.306813] LustreError: 283173:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4773.800744] LustreError: 284208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781687010 with bad export cookie 17251637079085034165 [ 4773.807199] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4773.808388] LustreError: 284208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4774.111599] Lustre: server umount lustre-MDT0001 complete [ 4781.323669] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 4783.466464] Key type lgssc unregistered [ 4783.718854] LNet: 286689:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4783.726431] LNetError: 286689:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4783.761959] LNet: Removed LNI 192.168.204.144@tcp [ 4784.456996] Key type .llcrypt unregistered [ 4784.459746] Key type ._llcrypt unregistered [ 4795.546267] Key type ._llcrypt registered [ 4795.548466] Key type .llcrypt registered [ 4795.609117] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 4796.206603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4796.216642] alg: No test for adler32 (adler32-zlib) [ 4797.188636] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 4797.346798] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 4798.983220] Key type lgssc registered [ 4799.757307] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4805.063282] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4805.081884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4806.461265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4809.467265] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4815.505455] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4815.761582] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4818.761319] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4823.415608] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4826.817265] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4830.759533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4830.972700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4831.977506] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4834.623553] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4838.652490] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4851.171753] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4851.179152] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4851.188553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4851.684602] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4851.685669] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4851.703682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4855.692428] Lustre: server umount lustre-OST0000 complete [ 4861.921831] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4861.926290] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4861.929934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4861.935075] Lustre: Skipped 1 previous similar message [ 4864.288210] Lustre: server umount lustre-MDT0000 complete [ 4866.590928] LustreError: 287870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781687103 with bad export cookie 13870082065721091957 [ 4866.594148] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4866.599904] LustreError: 287870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4866.825821] Lustre: server umount lustre-MDT0001 complete [ 4872.726542] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 4874.445067] Key type lgssc unregistered [ 4874.648265] LNet: 291602:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4874.657601] LNetError: 291602:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4874.676623] LNet: Removed LNI 192.168.204.144@tcp [ 4875.330492] Key type .llcrypt unregistered [ 4875.333514] Key type ._llcrypt unregistered [ 4881.851686] Key type ._llcrypt registered [ 4881.853639] Key type .llcrypt registered [ 4881.942689] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4893.328314] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 4903.084279] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 4903.670360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4903.745096] alg: No test for adler32 (adler32-zlib) [ 4904.750127] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 4904.940293] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 4906.591213] Key type lgssc registered [ 4907.439496] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4930.955364] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4930.977996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4932.165148] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4932.193790] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4932.248856] Lustre: lustre-MDT0000: new disk, initializing [ 4932.299973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4932.311989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4934.727341] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4941.527594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4941.608732] Lustre: 295130:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4941.655834] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4941.662803] Lustre: Skipped 1 previous similar message [ 4941.721639] Lustre: lustre-MDT0001: new disk, initializing [ 4941.787261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4941.821056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4941.829842] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4944.442397] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4950.407548] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4953.777367] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4957.874570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4958.104141] Lustre: lustre-OST0000: new disk, initializing [ 4958.107706] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4958.113413] Lustre: 297098:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4958.179638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4961.924812] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 4967.461931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4967.471578] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4967.559679] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4967.943569] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4970.499377] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4970.630390] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4973.439808] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4973.619296] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4977.635961] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4977.640205] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4977.642228] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4977.655453] Lustre: Skipped 1 previous similar message [ 4977.665891] Lustre: Skipped 1 previous similar message [ 4981.048534] Lustre: server umount lustre-OST0000 complete [ 4987.880065] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4987.881169] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4987.885142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4987.895631] Lustre: Skipped 1 previous similar message [ 4989.654445] Lustre: server umount lustre-MDT0000 complete [ 4992.399859] LustreError: 295119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781687229 with bad export cookie 823197501542398319 [ 4992.402076] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4992.409570] LustreError: 295119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4992.614342] Lustre: server umount lustre-MDT0001 complete [ 4998.055560] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 05:07:13 (1781687233) [ 5002.394915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5002.709160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5005.703753] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5007.840849] LustreError: 299509:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5007.855326] LustreError: 299509:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5011.132231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5011.371627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5013.721409] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5018.220647] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5021.739684] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5025.749859] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5025.961834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5029.679828] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5034.473535] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5041.633897] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5041.640196] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5041.647290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5041.651885] Lustre: Skipped 1 previous similar message [ 5041.676036] Lustre: Skipped 2 previous similar messages [ 5046.753766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5046.759048] Lustre: Skipped 1 previous similar message [ 5047.574206] Lustre: server umount lustre-OST0000 complete [ 5051.872979] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5051.875921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5051.879146] Lustre: Skipped 1 previous similar message [ 5052.071614] Lustre: server umount lustre-MDT0000 complete [ 5054.529741] LustreError: 299488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781687291 with bad export cookie 823197501542399194 [ 5054.533634] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5054.539030] LustreError: 299488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5054.697765] Lustre: server umount lustre-MDT0001 complete [ 5061.153911] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 5063.078392] Key type lgssc unregistered [ 5063.313237] LNet: 303041:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5063.320729] LNetError: 303041:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5063.331325] LNet: Removed LNI 192.168.204.144@tcp [ 5063.894753] Key type .llcrypt unregistered [ 5063.900775] Key type ._llcrypt unregistered [ 5080.053702] Key type ._llcrypt registered [ 5080.055805] Key type .llcrypt registered [ 5080.123693] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 5089.197365] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 5089.836602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5089.856308] alg: No test for adler32 (adler32-zlib) [ 5090.815329] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 5090.964970] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 5092.583201] Key type lgssc registered [ 5093.271609] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5120.207598] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5120.230349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5121.417577] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5121.448903] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5121.516436] Lustre: lustre-MDT0000: new disk, initializing [ 5121.579301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5121.600199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5123.791709] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5129.584362] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5129.642402] Lustre: 306841:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5129.675348] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5129.678424] Lustre: Skipped 1 previous similar message [ 5129.717735] Lustre: lustre-MDT0001: new disk, initializing [ 5129.777536] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5129.819721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5129.826487] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5131.873219] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5137.110107] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5140.416180] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5143.960796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5144.104856] Lustre: lustre-OST2710: new disk, initializing [ 5144.107806] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 5144.111879] Lustre: 308810:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5144.153438] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 5147.099609] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5150.735656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 5150.743325] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 5150.822268] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 5151.786228] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 5154.889536] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5154.992617] Lustre: lustre-OST03e8: new disk, initializing [ 5154.995963] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 5155.000479] Lustre: 309875:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5155.050464] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 5156.236762] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 5156.245606] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 5156.310885] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 5157.850131] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5161.220745] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 5165.603607] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 5165.704275] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5167.544672] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 5167.639364] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5169.595747] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 5169.703375] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5171.680901] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 5171.787285] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5175.776094] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5175.779775] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5175.785682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5176.802749] Lustre: lustre-MDT0000-lwp-OST03e8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5176.803174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5176.813097] Lustre: Skipped 2 previous similar messages [ 5176.816055] Lustre: Skipped 2 previous similar messages [ 5179.452186] Lustre: server umount lustre-MDT0000 complete [ 5181.087522] LustreError: 306832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781687417 with bad export cookie 16806486078287440823 [ 5181.091469] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5181.093067] LustreError: 306832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5181.242414] Lustre: server umount lustre-MDT0001 complete [ 5193.236513] Lustre: server umount lustre-OST2710 complete [ 5205.222639] Lustre: server umount lustre-OST03e8 complete [ 5212.646739] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 5216.843888] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 5232.742555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5232.860906] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5232.877743] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5232.922143] Lustre: lustre-MDT0000: new disk, initializing [ 5232.966945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5232.984211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5234.733571] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5239.838517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5239.885573] Lustre: 315115:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5240.000607] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5241.717493] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5246.110324] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5248.209937] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5250.937594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5251.059781] Lustre: lustre-OST0000: new disk, initializing [ 5251.062135] Lustre: Skipped 1 previous similar message [ 5251.064880] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5251.067507] Lustre: Skipped 2 previous similar messages [ 5251.070685] Lustre: 317083:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5251.106145] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5251.109074] Lustre: Skipped 1 previous similar message [ 5253.361633] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5257.651116] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5259.377453] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5261.312653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5261.318079] Lustre: Skipped 1 previous similar message [ 5261.321150] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5261.369494] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5261.515627] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 5263.361112] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5263.462381] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5266.401922] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5266.402302] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5266.408154] LustreError: Skipped 1 previous similar message [ 5266.409253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5266.413058] Lustre: Skipped 1 previous similar message [ 5270.511591] Lustre: server umount lustre-OST0000 complete [ 5276.128147] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5276.132060] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5276.139928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5276.142616] Lustre: Skipped 1 previous similar message [ 5278.514038] Lustre: server umount lustre-MDT0000 complete [ 5280.171294] LustreError: 315108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781687516 with bad export cookie 16806486078287442503 [ 5280.174820] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5280.177755] LustreError: 315108:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5280.335898] Lustre: server umount lustre-MDT0001 complete [ 5284.004446] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 05:11:59 (1781687519) [ 5284.678371] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 5285.410632] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 05:12:01 (1781687521) [ 5286.753709] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 5298.809202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5298.968880] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5298.987642] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5299.083602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5301.145340] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5305.688218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5305.745404] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5305.764775] Lustre: 320948:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5307.898498] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5325.092436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5325.196959] Lustre: Denying initial registration attempt from nid 192.168.204.144@tcp, specified as failover [ 5325.202372] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 5325.206663] LustreError: 322402:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -99 [ 5325.211034] LustreError: 322402:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 5325.214843] LustreError: 322402:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 5325.250992] Lustre: server umount lustre-OST0000 complete [ 5325.253073] LustreError: 322402:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 5331.424544] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5331.431771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5331.438926] Lustre: Skipped 1 previous similar message [ 5331.442395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5331.445865] Lustre: Skipped 1 previous similar message [ 5334.561235] Lustre: server umount lustre-MDT0000 complete [ 5336.114581] LustreError: 322770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781687572 with bad export cookie 16806486078287443371 [ 5336.117257] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5336.119659] LustreError: 322770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5336.260754] Lustre: server umount lustre-MDT0001 complete [ 5340.718973] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 5342.043793] Key type lgssc unregistered [ 5342.190575] LNet: 323522:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5342.194737] LNetError: 323522:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5342.205426] LNet: Removed LNI 192.168.204.144@tcp [ 5342.568358] Key type .llcrypt unregistered [ 5342.570295] Key type ._llcrypt unregistered [ 5347.426313] Key type ._llcrypt registered [ 5347.427920] Key type .llcrypt registered [ 5347.474773] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 5347.856822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5347.919495] alg: No test for adler32 (adler32-zlib) [ 5348.810794] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 5348.919164] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 5350.527200] Key type lgssc registered [ 5350.988141] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5379.447676] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 5383.408453] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5383.417288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5384.559132] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5384.572909] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5384.660482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5386.378640] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5390.403834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5390.465266] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5390.490390] Lustre: 326262:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5390.636850] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5392.517089] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5395.859071] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5398.256935] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5414.308163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5414.412285] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5414.509558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5417.192209] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5420.656131] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5421.711702] Lustre: server umount lustre-OST0000 complete [ 5426.656169] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5426.659834] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5426.667090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5427.168933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5427.169546] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5429.608605] Lustre: server umount lustre-MDT0000 complete [ 5431.281223] LustreError: 326254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781687668 with bad export cookie 16079378369038769855 [ 5431.284154] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5431.287051] LustreError: 326254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5431.425330] Lustre: server umount lustre-MDT0001 complete [ 5436.159630] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 5437.453208] Key type lgssc unregistered [ 5437.595462] LNet: 329829:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5437.601862] LNetError: 329829:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5437.614651] LNet: Removed LNI 192.168.204.144@tcp [ 5437.972559] Key type .llcrypt unregistered [ 5437.974222] Key type ._llcrypt unregistered [ 5449.677197] Key type ._llcrypt registered [ 5449.679842] Key type .llcrypt registered [ 5449.731277] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 5450.102973] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5450.151917] alg: No test for adler32 (adler32-zlib) [ 5451.051148] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 5451.181236] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 5452.791211] Key type lgssc registered [ 5453.306662] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5457.188102] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5457.198938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5458.413490] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5460.279802] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5464.313679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5464.535311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5466.414084] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5469.663582] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5472.127222] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5477.219420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5477.380793] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5479.927512] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5482.803991] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5499.666349] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.44@tcp (stopping) [ 5499.873831] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5499.878894] Lustre: Skipped 2 previous similar messages [ 5503.266620] Lustre: server umount lustre-MDT0000 complete [ 5504.664395] LustreError: 331205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781687741 with bad export cookie 5740692561590557098 [ 5504.667522] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5504.670172] LustreError: 331205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5504.778577] LustreError: 331220:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5504.785598] LustreError: 331220:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5504.789420] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.44@tcp (stopping) [ 5504.792092] Lustre: Skipped 3 previous similar messages [ 5504.992472] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5509.897997] LustreError: 333661:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5509.898097] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.44@tcp (stopping) [ 5509.909427] LustreError: 333661:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5509.917363] Lustre: Skipped 1 previous similar message [ 5510.852561] Lustre: server umount lustre-MDT0001 complete [ 5513.481205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5516.909574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5529.570230] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x4fab09782255359c [ 5529.575234] Lustre: MGC192.168.204.144@tcp: Connection restored to 0@lo (at 0@lo) [ 5529.701057] LustreError: 333052:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5529.709078] LustreError: 333052:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5529.734144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5529.757070] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 5531.356432] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5534.880534] LustreError: 334833:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5534.887712] LustreError: 334833:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5535.255757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5535.410036] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5537.295518] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5539.496447] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5540.680605] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5541.445374] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5544.050307] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5544.804584] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5550.055219] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5550.055915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5550.060027] Lustre: Skipped 1 previous similar message [ 5550.066387] Lustre: Skipped 2 previous similar messages [ 5552.737122] Lustre: server umount lustre-OST0000 complete [ 5555.171807] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5555.175898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5555.177294] Lustre: Skipped 1 previous similar message [ 5555.184139] Lustre: Skipped 1 previous similar message [ 5560.484245] Lustre: server umount lustre-MDT0000 complete [ 5561.920488] LustreError: 334022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781687798 with bad export cookie 5740692561590564252 [ 5561.925886] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5561.926546] LustreError: 334022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5562.049873] Lustre: server umount lustre-MDT0001 complete [ 5566.220362] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 5567.332323] Key type lgssc unregistered [ 5567.471454] LNet: 337500:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5567.474739] LNetError: 337500:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5567.485365] LNet: Removed LNI 192.168.204.144@tcp [ 5567.814824] Key type .llcrypt unregistered [ 5567.815906] Key type ._llcrypt unregistered [ 5578.555353] Key type ._llcrypt registered [ 5578.557462] Key type .llcrypt registered [ 5578.607903] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 5578.972866] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5579.021661] alg: No test for adler32 (adler32-zlib) [ 5579.903611] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 5580.009666] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 5581.607130] Key type lgssc registered [ 5582.070662] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5585.427409] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5585.434691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5586.655805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5588.250784] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5591.912165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5592.071351] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5593.856993] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5596.676678] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5598.838786] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5601.249666] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5601.254025] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5601.258926] Lustre: Skipped 1 previous similar message [ 5602.272655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5602.275941] Lustre: Skipped 2 previous similar messages [ 5606.182980] Lustre: server umount lustre-MDT0000 complete [ 5607.393186] LustreError: 339623:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5607.400899] LustreError: 339623:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5607.718141] LustreError: 338875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781687844 with bad export cookie 3025712867743635953 [ 5607.723340] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5607.853557] Lustre: server umount lustre-MDT0001 complete [ 5613.443052] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5613.599523] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5613.616767] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5613.723782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5615.346949] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5618.519630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5618.565924] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5618.580887] Lustre: 341481:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5618.691502] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5620.305947] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5623.030942] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5625.140573] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5630.179031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5630.265224] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 5630.271149] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 5630.277496] LustreError: 343365:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 5630.281795] LustreError: 343365:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 5630.285598] LustreError: 343365:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 5630.314486] Lustre: server umount lustre-OST0000 complete [ 5630.316625] LustreError: 343365:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5635.427543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5635.514501] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5635.600935] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5637.918149] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5640.959977] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5641.244208] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 5644.286497] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5644.320156] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5644.359643] Lustre: lustre-OST0001: new disk, initializing [ 5644.361755] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5644.365182] Lustre: 344902:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5644.404428] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5646.961587] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5650.117111] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5650.954982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5650.959877] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5651.000343] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5656.037572] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5656.038556] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5656.042421] LustreError: Skipped 1 previous similar message [ 5656.048873] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5656.052037] Lustre: Skipped 1 previous similar message [ 5657.412809] Lustre: server umount lustre-OST0001 complete [ 5661.155360] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5661.155400] LustreError: 343832:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5661.156344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5661.160223] Lustre: Skipped 2 previous similar messages [ 5661.166331] LustreError: 343832:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5665.046067] Lustre: server umount lustre-OST0000 complete [ 5669.856236] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5669.859762] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5669.864712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5669.867171] Lustre: Skipped 1 previous similar message [ 5672.615492] Lustre: server umount lustre-MDT0000 complete [ 5674.161285] LustreError: 341472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781687910 with bad export cookie 3025712867743636485 [ 5674.164592] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5674.166158] LustreError: 341472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 5674.317845] Lustre: server umount lustre-MDT0001 complete [ 5678.524301] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 5679.754470] Key type lgssc unregistered [ 5679.904032] LNet: 346859:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5679.908653] LNetError: 346859:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5679.918652] LNet: Removed LNI 192.168.204.144@tcp [ 5680.292246] Key type .llcrypt unregistered [ 5680.293708] Key type ._llcrypt unregistered [ 5714.718595] Key type ._llcrypt registered [ 5714.720051] Key type .llcrypt registered [ 5714.765786] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 5721.730252] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 5722.130465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5722.140223] alg: No test for adler32 (adler32-zlib) [ 5723.015169] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 5723.110328] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 5724.695161] Key type lgssc registered [ 5725.188150] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5738.326862] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5738.333906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5739.453546] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5739.472183] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5739.516667] Lustre: lustre-MDT0000: new disk, initializing [ 5739.558221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5739.566080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5741.149125] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5745.873951] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5745.929312] Lustre: 351760:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5745.956696] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5745.959392] Lustre: Skipped 1 previous similar message [ 5745.999211] Lustre: lustre-MDT0001: new disk, initializing [ 5746.039475] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5746.054457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5746.059118] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5747.633092] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5751.834879] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5753.877285] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5756.313347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5756.421764] Lustre: lustre-OST0000: new disk, initializing [ 5756.424471] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5756.428553] Lustre: 353729:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5756.460411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5758.547836] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5762.238408] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5762.552583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5762.556122] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5762.592915] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5763.675143] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5763.749082] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5765.177633] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5765.257827] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5767.649862] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5767.650432] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5767.653397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5767.656801] Lustre: Skipped 1 previous similar message [ 5772.062885] Lustre: server umount lustre-OST0000 complete [ 5776.863929] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5776.867204] LustreError: Skipped 1 previous similar message [ 5776.869243] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5776.875186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5776.878316] Lustre: Skipped 1 previous similar message [ 5777.888551] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5777.888866] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5779.549785] Lustre: server umount lustre-MDT0000 complete [ 5781.014201] LustreError: 351752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781688017 with bad export cookie 5396698936948895031 [ 5781.017288] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5781.019297] LustreError: 351752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5781.158608] Lustre: server umount lustre-MDT0001 complete [ 5784.184935] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 05:20:20 (1781688020) [ 5787.070452] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 05:20:22 (1781688022) [ 5790.660950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5790.905557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5792.600920] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5796.217235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5796.321111] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5796.324714] Lustre: Skipped 1 previous similar message [ 5796.375596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5798.113425] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5800.808309] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5802.799582] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5805.729378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5805.899400] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5808.558364] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5811.614477] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5820.559763] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 5821.318968] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5821.981526] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5822.629370] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 5823.380771] Lustre: Failing over lustre-MDT0000 [ 5823.591273] Lustre: server umount lustre-MDT0000 complete [ 5826.530131] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5826.531532] LustreError: 356475:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5826.535397] Lustre: Skipped 2 previous similar messages [ 5826.543396] LustreError: 356475:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5827.347686] LustreError: 356475:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5831.649750] LustreError: 356479:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5831.656402] LustreError: 356479:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5836.769259] LustreError: 357217:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5836.778914] LustreError: 357217:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5837.019529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5837.064878] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5837.165836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5837.194882] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5837.579306] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5838.871104] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5842.405813] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5842.426675] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 5842.454321] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5844.660463] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5845.291827] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5847.287258] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 5848.019200] Lustre: Failing over lustre-MDT0000 [ 5848.284211] Lustre: server umount lustre-MDT0000 complete [ 5852.641413] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5852.641716] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5852.642760] LustreError: 357217:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5852.646765] Lustre: Skipped 2 previous similar messages [ 5861.708992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5861.769430] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5861.907295] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5863.180452] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5863.580694] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5866.984834] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5866.987966] Lustre: Skipped 2 previous similar messages [ 5867.000097] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 5867.022039] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5869.066748] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5869.676790] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5872.097393] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5872.098104] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5872.098731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5872.101876] Lustre: Skipped 2 previous similar messages [ 5878.240483] Lustre: server umount lustre-MDT0000 complete [ 5879.680488] LustreError: 356458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781688116 with bad export cookie 5396698936948897712 [ 5879.683546] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5879.685409] LustreError: 356458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5879.804567] Lustre: server umount lustre-MDT0001 complete [ 5891.563149] Lustre: server umount lustre-OST0000 complete [ 5897.121606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5897.302537] LustreError: 362809:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5897.310443] LustreError: 362809:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 5897.327747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5897.330909] Lustre: Skipped 1 previous similar message [ 5898.851219] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5902.091872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5903.851159] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5906.565105] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5908.694392] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5911.374521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5913.911298] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5916.658815] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5922.850083] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5933.025672] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5933.027306] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5933.028992] LustreError: Skipped 1 previous similar message [ 5933.036423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5933.040483] Lustre: Skipped 6 previous similar messages [ 5934.674151] Lustre: server umount lustre-OST0000 complete [ 5942.172760] Lustre: server umount lustre-MDT0000 complete [ 5943.264654] LustreError: 362809:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5943.272467] LustreError: 362809:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5943.542205] LustreError: 363544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781688180 with bad export cookie 5396698936948898461 [ 5943.545625] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5943.546580] LustreError: 363544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5943.662631] Lustre: server umount lustre-MDT0001 complete [ 5947.602860] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 5948.722250] Key type lgssc unregistered [ 5948.853468] LNet: 366362:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5948.856843] LNetError: 366362:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5948.863330] LNet: Removed LNI 192.168.204.144@tcp [ 5949.176462] Key type .llcrypt unregistered [ 5949.178095] Key type ._llcrypt unregistered [ 5959.857827] Key type ._llcrypt registered [ 5959.859547] Key type .llcrypt registered [ 5959.900539] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 5960.249278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5960.272118] alg: No test for adler32 (adler32-zlib) [ 5961.152682] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 5961.237951] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 5962.832976] Key type lgssc registered [ 5963.363536] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5966.512481] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5966.519607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5967.733750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5969.370187] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5972.796172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5972.943721] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5974.505101] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5977.039394] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5978.953886] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5981.486337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5981.630620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5983.883123] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 5986.719350] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5992.940295] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5998.052055] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5998.053223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5998.056689] Lustre: Skipped 1 previous similar message [ 5998.060776] Lustre: Skipped 2 previous similar messages [ 6003.170617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6003.173602] Lustre: Skipped 2 previous similar messages [ 6003.222617] Lustre: server umount lustre-MDT0000 complete [ 6004.582551] LustreError: 368590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781688241 with bad export cookie 9405752306588522902 [ 6004.585347] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6004.586971] LustreError: 368590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6004.720043] Lustre: server umount lustre-MDT0001 complete [ 6015.327525] Lustre: server umount lustre-OST0000 complete [ 6020.213433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6020.402618] LustreError: 371565:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6020.410774] LustreError: 371565:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6020.440876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6021.990696] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6025.280821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6025.402502] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6026.867607] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6029.491816] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6031.485373] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6034.175854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6034.296105] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6036.548871] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6039.232366] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6045.542285] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 6056.415806] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6056.419365] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6056.423920] Lustre: Skipped 1 previous similar message [ 6056.426163] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6061.024710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6061.026451] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6061.028941] Lustre: Skipped 1 previous similar message [ 6061.034257] Lustre: Skipped 1 previous similar message [ 6062.425340] Lustre: server umount lustre-MDT0000 complete [ 6063.603992] LustreError: 371545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781688300 with bad export cookie 9405752306588524001 [ 6063.607030] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6063.608404] LustreError: 371545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6063.718817] Lustre: server umount lustre-MDT0001 complete [ 6074.261183] Lustre: server umount lustre-OST0000 complete [ 6079.381658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6079.520251] LustreError: 375494:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6079.540298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6080.932805] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6083.742908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6085.078840] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6087.406253] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6089.158955] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6091.403338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6093.450106] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6095.941152] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6105.829465] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 6109.151684] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6109.154623] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6109.159713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6109.161461] Lustre: Skipped 1 previous similar message [ 6111.202640] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6113.745287] Lustre: server umount lustre-OST0000 complete [ 6116.320493] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6116.324464] Lustre: Skipped 1 previous similar message [ 6121.173549] Lustre: server umount lustre-MDT0000 complete [ 6121.440112] LustreError: 377983:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6121.446541] LustreError: 377983:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6122.488247] LustreError: 375474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781688359 with bad export cookie 9405752306588525009 [ 6122.490891] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6122.493726] LustreError: 375474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6122.615280] Lustre: server umount lustre-MDT0001 complete [ 6126.614547] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 6127.706895] Key type lgssc unregistered [ 6127.844420] LNet: 379044:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6127.848502] LNetError: 379044:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6127.857307] LNet: Removed LNI 192.168.204.144@tcp [ 6128.152429] Key type .llcrypt unregistered [ 6128.154056] Key type ._llcrypt unregistered [ 6137.843738] Key type ._llcrypt registered [ 6137.844714] Key type .llcrypt registered [ 6137.882741] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 6138.225663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6138.262160] alg: No test for adler32 (adler32-zlib) [ 6139.119450] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 6139.205419] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 6140.791163] Key type lgssc registered [ 6141.219258] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6143.996798] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6144.004950] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6144.009398] LustreError: 380462:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 6144.019342] LustreError: 380462:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 6144.022726] LustreError: 380462:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 6144.025971] LustreError: 380462:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 6144.029527] LustreError: 380462:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6146.690612] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6146.693676] LustreError: 380858:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 6146.702117] LustreError: 380858:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 6146.705176] LustreError: 380858:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 6146.708749] LustreError: 380858:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 6146.711222] LustreError: 380858:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6153.303595] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 6154.377139] Key type lgssc unregistered [ 6154.525360] LNet: 381859:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6154.528421] LNetError: 381859:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6154.539341] LNet: Removed LNI 192.168.204.144@tcp [ 6154.850757] Key type .llcrypt unregistered [ 6154.852324] Key type ._llcrypt unregistered [ 6161.170404] Key type ._llcrypt registered [ 6161.172013] Key type .llcrypt registered [ 6161.221067] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 6167.584229] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 6167.940886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6168.006072] alg: No test for adler32 (adler32-zlib) [ 6168.874802] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 6168.963721] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 6170.551172] Key type lgssc registered [ 6170.928511] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6182.894333] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6182.902072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6183.989783] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6184.000559] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6184.035405] Lustre: lustre-MDT0000: new disk, initializing [ 6184.064081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6184.070924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6185.474767] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6189.711866] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6189.756948] Lustre: 384987:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6189.785206] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6189.787959] Lustre: Skipped 1 previous similar message [ 6189.825571] Lustre: lustre-MDT0001: new disk, initializing [ 6189.863651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6189.885626] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6189.890710] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6191.316169] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6195.115363] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6197.043370] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6199.409892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6199.507560] Lustre: lustre-OST0000: new disk, initializing [ 6199.509543] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6199.512205] Lustre: 386956:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6199.542621] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6201.519470] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6205.222870] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6206.628357] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6206.966298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6206.969460] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6206.997512] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6207.714961] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 6209.092016] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6209.164721] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6212.067321] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6212.069730] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6212.070043] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6212.072083] Lustre: Skipped 1 previous similar message [ 6215.890974] Lustre: server umount lustre-OST0000 complete [ 6217.184213] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6217.184558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6217.189352] Lustre: Skipped 1 previous similar message [ 6217.190982] Lustre: Skipped 1 previous similar message [ 6222.304115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6222.307353] Lustre: Skipped 2 previous similar messages [ 6223.259820] Lustre: server umount lustre-MDT0000 complete [ 6224.514323] LustreError: 385785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781688461 with bad export cookie 4135661412382969213 [ 6224.517738] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6224.518679] LustreError: 385785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6224.636544] Lustre: server umount lustre-MDT0001 complete [ 6227.456769] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 6228.019793] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 05:27:44 (1781688464) [ 6230.238325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6230.406359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6231.805472] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6234.645447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6234.771987] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6236.167742] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6238.673989] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6240.492577] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6242.803244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6242.916113] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6244.767110] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6247.078825] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6249.418465] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6249.479388] Lustre: lustre-OST0001: new disk, initializing [ 6249.481646] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6249.484888] Lustre: 392256:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6251.364425] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6255.025222] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6257.662727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6257.666741] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6257.691632] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6262.752857] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6262.755513] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6262.756555] LustreError: Skipped 1 previous similar message [ 6262.762246] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6262.765136] Lustre: Skipped 1 previous similar message [ 6263.227749] Lustre: server umount lustre-OST0001 complete [ 6263.864211] LustreError: 391242:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6263.871282] LustreError: 391242:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6267.877779] LustreError: 392973:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6267.883929] LustreError: 392973:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6272.995378] LustreError: 392973:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6272.999867] LustreError: 392973:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6276.064291] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6276.066707] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6276.070132] Lustre: Skipped 1 previous similar message [ 6276.071583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6278.114165] LustreError: 391242:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6278.118703] LustreError: 391242:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6283.234263] LustreError: 391243:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6283.239569] LustreError: 391243:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6288.353030] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6288.354768] Lustre: Skipped 6 previous similar messages [ 6289.375111] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6289.422307] Lustre: server umount lustre-OST0000 complete [ 6291.423826] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6291.426531] LustreError: Skipped 1 previous similar message [ 6291.428041] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6291.431630] Lustre: Skipped 1 previous similar message [ 6296.792328] Lustre: server umount lustre-MDT0000 complete [ 6297.985089] LustreError: 389403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781688534 with bad export cookie 4135661412382970095 [ 6297.988358] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6297.990783] LustreError: 389403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6304.173404] Lustre: server umount lustre-MDT0001 complete [ 6307.690918] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 6308.664572] Key type lgssc unregistered [ 6308.784337] LNet: 394245:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6308.787145] LNetError: 394245:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6308.794358] LNet: Removed LNI 192.168.204.144@tcp [ 6309.105786] Key type .llcrypt unregistered [ 6309.107327] Key type ._llcrypt unregistered [ 6327.005758] Key type ._llcrypt registered [ 6327.007251] Key type .llcrypt registered [ 6327.040039] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 6327.409480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6327.420309] alg: No test for adler32 (adler32-zlib) [ 6328.265425] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 6328.343901] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 6329.927167] Key type lgssc registered [ 6330.260902] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6332.786305] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6332.793130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6333.878438] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6333.888161] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6333.954034] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6335.191254] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6337.868263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6337.904255] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6337.916190] Lustre: 397019:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6338.005769] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6339.224441] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6341.474394] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6343.101475] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6344.672397] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6344.673019] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6344.675960] Lustre: Skipped 1 previous similar message [ 6349.792182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6349.794349] Lustre: Skipped 2 previous similar messages [ 6349.906295] Lustre: server umount lustre-MDT0000 complete [ 6351.083125] LustreError: 397009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781688587 with bad export cookie 9894656131110609191 [ 6351.085729] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6351.086608] LustreError: 397009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6351.197680] Lustre: server umount lustre-MDT0001 complete [ 6354.232183] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 05:29:50 (1781688590) [ 6356.314973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6356.470285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6357.699251] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6360.516586] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6360.631409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6361.913495] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6364.283217] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6366.029606] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6368.086163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6368.155411] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6368.211150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6369.943632] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6372.200375] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6375.828756] Lustre: Permanently deactivating lustre-OST0000 [ 6375.833311] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6377.197076] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6377.651348] LustreError: 402366:0:(mgs_llog.c:2041:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 6377.654399] LustreError: 402366:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [ 6377.656693] LustreError: 402366:0:(mgs_handler.c:1245:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6378.183567] LustreError: 402415:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [ 6378.186346] LustreError: 402415:0:(mgs_handler.c:1245:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6384.911333] Lustre: server umount lustre-OST0000 complete [ 6385.800211] LustreError: 402664:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [ 6385.804059] LustreError: 402664:0:(mgs_handler.c:1245:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6390.752560] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6390.753110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6390.756124] Lustre: Skipped 1 previous similar message [ 6395.871908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6395.873821] Lustre: Skipped 2 previous similar messages [ 6396.919256] Lustre: server umount lustre-MDT0000 complete [ 6398.045307] LustreError: 399553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781688634 with bad export cookie 9894656131110609765 [ 6398.048133] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6398.050361] LustreError: 399553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6407.695562] Lustre: server umount lustre-MDT0001 complete [ 6410.467525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6411.790099] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6412.685630] LustreError: 403819:0:(mgs_handler.c:1245:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 6414.751484] Lustre: 404157:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.144@tcp [ 6415.456144] Lustre: server umount MGS complete [ 6419.281071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6419.436048] LustreError: 404880:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6419.441311] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6419.441559] LustreError: 404880:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6419.443495] Lustre: Skipped 1 previous similar message [ 6419.467176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6420.862826] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6423.786726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6423.887543] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6425.304396] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6427.775275] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6429.632938] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6430.298614] Lustre: Permanently reactivating lustre-OST0000 [ 6430.305416] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6430.307587] Lustre: Skipped 2 previous similar messages [ 6455.263286] LustreError: 406661:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 6458.622258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6458.752444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6458.754603] Lustre: Skipped 1 previous similar message [ 6460.626121] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6463.003786] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6475.232427] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6475.234993] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6475.239276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6476.258753] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6476.876101] Lustre: server umount lustre-OST0000 complete [ 6480.222795] Lustre: server umount lustre-MDT0000 complete [ 6480.351772] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6480.354248] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6480.360256] Lustre: Skipped 1 previous similar message [ 6480.362915] LustreError: 404899:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6481.376171] LustreError: 407001:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6481.380828] LustreError: 407001:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6481.459141] LustreError: 404861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781688718 with bad export cookie 9894656131110610787 [ 6481.462103] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6481.462929] LustreError: 404861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6481.580744] Lustre: server umount lustre-MDT0001 complete [ 6485.422834] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 6486.427737] Key type lgssc unregistered [ 6486.557416] LNet: 409550:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6486.560946] LNetError: 409550:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6486.570330] LNet: Removed LNI 192.168.204.144@tcp [ 6486.864348] Key type .llcrypt unregistered [ 6486.865913] Key type ._llcrypt unregistered [ 6492.879674] Key type ._llcrypt registered [ 6492.880865] Key type .llcrypt registered [ 6492.920135] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 6499.102210] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 6499.431251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6499.464901] alg: No test for adler32 (adler32-zlib) [ 6500.308681] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 6500.379889] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 6501.959076] Key type lgssc registered [ 6502.332986] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6513.691043] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6513.697317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6514.789951] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6514.801217] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6514.838077] Lustre: lustre-MDT0000: new disk, initializing [ 6514.864413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6514.872525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6516.179142] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6520.254771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6520.302503] Lustre: 412678:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6520.324632] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6520.326580] Lustre: Skipped 1 previous similar message [ 6520.365119] Lustre: lustre-MDT0001: new disk, initializing [ 6520.403056] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6520.417441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6520.421776] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6521.807061] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6525.468173] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6527.299989] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6529.594352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6529.697298] Lustre: lustre-OST0000: new disk, initializing [ 6529.699982] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6529.702815] Lustre: 414649:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6529.737394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6531.746750] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6532.982248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6532.987470] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6533.015373] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6535.457255] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6536.903947] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6536.988311] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6538.367568] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6538.440103] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6543.330491] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6543.330793] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6543.331427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6543.334168] Lustre: Skipped 1 previous similar message [ 6543.336167] LustreError: Skipped 1 previous similar message [ 6545.168347] Lustre: server umount lustre-OST0000 complete [ 6548.448609] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6548.449335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6548.453028] Lustre: Skipped 1 previous similar message [ 6548.455083] Lustre: Skipped 1 previous similar message [ 6552.536057] Lustre: server umount lustre-MDT0000 complete [ 6553.801202] LustreError: 412672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781688790 with bad export cookie 1977199677937427435 [ 6553.804992] LustreError: 412672:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6553.805706] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6553.925530] Lustre: server umount lustre-MDT0001 complete [ 6556.868711] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 6557.387884] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 05:33:13 (1781688793) [ 6559.504164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6559.643027] LustreError: 417106:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6559.648726] LustreError: 417106:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6559.664029] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6560.803498] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6563.340796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6564.757792] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6567.133021] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6568.900324] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6571.205399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6571.336687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6571.338668] Lustre: Skipped 1 previous similar message [ 6573.259436] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6575.652210] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6576.332706] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 6587.221111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 6588.895901] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6588.898245] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6588.903351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6589.922233] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6594.766360] Lustre: server umount lustre-OST0000 complete [ 6599.647854] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6599.650432] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6599.654525] Lustre: Skipped 1 previous similar message [ 6599.656912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6599.659718] Lustre: Skipped 2 previous similar messages [ 6602.137109] Lustre: server umount lustre-MDT0000 complete [ 6603.366464] LustreError: 417087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781688840 with bad export cookie 1977199677937428310 [ 6603.369559] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6603.371252] LustreError: 417087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6603.493388] Lustre: server umount lustre-MDT0001 complete [ 6607.328717] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 6608.374981] Key type lgssc unregistered [ 6608.503335] LNet: 420849:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6608.505639] LNetError: 420849:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6608.516270] LNet: Removed LNI 192.168.204.144@tcp [ 6608.806927] Key type .llcrypt unregistered [ 6608.808263] Key type ._llcrypt unregistered [ 6616.532039] Key type ._llcrypt registered [ 6616.533388] Key type .llcrypt registered [ 6616.573674] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 6616.674480] Key type .llcrypt unregistered [ 6616.676119] Key type ._llcrypt unregistered [ 6623.798985] Key type ._llcrypt registered [ 6623.800223] Key type .llcrypt registered [ 6623.838324] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 6624.148055] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6624.157955] alg: No test for adler32 (adler32-zlib) [ 6625.022093] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 6625.113748] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 6626.695157] Key type lgssc registered [ 6627.068203] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6629.765957] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6629.772907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6630.934205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6632.387558] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6635.273771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6635.418881] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6637.362447] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6639.630204] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6640.544466] LustreError: 423037:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6640.550223] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6640.552848] LustreError: 423037:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6641.985871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6642.111980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6643.526139] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6650.847883] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6650.850427] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6650.854387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6652.384641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6652.385648] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6652.390344] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6656.845241] Lustre: server umount lustre-OST0000 complete [ 6662.113341] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 6662.117090] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6662.121841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6662.125167] Lustre: Skipped 1 previous similar message [ 6664.217816] Lustre: server umount lustre-MDT0000 complete [ 6665.472329] LustreError: 423016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781688902 with bad export cookie 18330569316032662409 [ 6665.474959] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6665.479100] LustreError: 423016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6665.609225] Lustre: server umount lustre-MDT0001 complete [ 6669.406358] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 6670.487684] Key type lgssc unregistered [ 6670.627433] LNet: 426184:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6670.630376] LNetError: 426184:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6670.638339] LNet: Removed LNI 192.168.204.144@tcp [ 6670.966633] Key type .llcrypt unregistered [ 6670.967892] Key type ._llcrypt unregistered [ 6679.523949] Key type ._llcrypt registered [ 6679.525734] Key type .llcrypt registered [ 6679.561949] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 6679.957439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6679.966690] alg: No test for adler32 (adler32-zlib) [ 6680.801571] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 6680.877274] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 6682.455141] Key type lgssc registered [ 6682.817146] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6685.428633] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6685.434531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6700.895378] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6706.015168] LustreError: 427550:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.144@tcp: failed processing log, type 4: rc = -110 [ 6732.639158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6734.536314] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6736.873724] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6739.059296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6739.231582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6739.252855] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6740.500489] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6743.441863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6743.589480] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6744.980799] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6747.445577] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6749.263651] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6752.739121] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6752.744610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6752.747093] Lustre: Skipped 1 previous similar message [ 6757.651250] Lustre: server umount lustre-OST0000 complete [ 6761.953451] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 6761.956248] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6761.960187] Lustre: Skipped 1 previous similar message [ 6761.962165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6762.976208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6762.976819] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6765.018854] Lustre: server umount lustre-MDT0000 complete [ 6766.331148] LustreError: 430076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781689003 with bad export cookie 11494747212313553703 [ 6766.334691] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6766.335971] LustreError: 430076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6766.440950] Lustre: server umount lustre-MDT0001 complete [ 6769.761822] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 6770.763903] Key type lgssc unregistered [ 6770.896404] LNet: 431113:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6770.899169] LNetError: 431113:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6770.908319] LNet: Removed LNI 192.168.204.144@tcp [ 6771.202764] Key type .llcrypt unregistered [ 6771.203791] Key type ._llcrypt unregistered [ 6779.697263] Key type ._llcrypt registered [ 6779.698836] Key type .llcrypt registered [ 6779.731686] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 6780.040206] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6780.070768] alg: No test for adler32 (adler32-zlib) [ 6780.916661] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 6780.993698] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 6782.567130] Key type lgssc registered [ 6782.904138] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6785.273333] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6785.280023] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6786.421253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6787.635084] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6790.266370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6790.373777] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6791.601838] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6793.730841] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6795.400910] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6797.687783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6797.800324] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6799.810385] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6802.274987] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6812.138608] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6816.224314] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6816.227129] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6816.231772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6817.248453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6817.250227] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6817.254819] Lustre: Skipped 1 previous similar message [ 6820.059937] Lustre: server umount lustre-MDT0000 complete [ 6820.825303] LustreError: 433239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.44@tcp arrived at 1781689057 with bad export cookie 13798443332476988879 [ 6821.484151] LustreError: 432485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781689058 with bad export cookie 13798443332476988228 [ 6821.487789] LustreError: 432485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6821.488160] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6822.367936] LustreError: 432506:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6822.368469] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6822.372758] LustreError: 432506:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6822.377409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6822.382612] Lustre: Skipped 3 previous similar messages [ 6826.399818] LustreError: 433245:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6826.400095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6826.405328] LustreError: 433245:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6827.600909] Lustre: server umount lustre-OST0000 complete [ 6829.918181] Lustre: server umount lustre-MDT0001 complete [ 6833.719822] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 6834.770405] Key type lgssc unregistered [ 6834.894447] LNet: 436208:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6834.897857] LNetError: 436208:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6834.906329] LNet: Removed LNI 192.168.204.144@tcp [ 6835.218342] Key type .llcrypt unregistered [ 6835.219790] Key type ._llcrypt unregistered [ 6844.651393] Key type ._llcrypt registered [ 6844.652459] Key type .llcrypt registered [ 6844.692472] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 6845.069863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6845.124969] alg: No test for adler32 (adler32-zlib) [ 6845.982489] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 6846.064707] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 6847.647204] Key type lgssc registered [ 6848.019474] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6850.583309] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6850.589904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6851.763166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6853.139794] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6856.038431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6856.167464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6857.523712] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6859.917193] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6861.689455] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6863.938158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6864.053613] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6865.936837] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6868.265949] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6876.326590] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6881.761163] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6881.764445] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6881.769059] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6881.771140] Lustre: Skipped 1 previous similar message [ 6884.113207] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.44@tcp (stopping) [ 6886.037607] Lustre: server umount lustre-MDT0001 complete [ 6886.880476] LustreError: 437601:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6886.888074] LustreError: 437601:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6892.000758] LustreError: 437602:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6892.001263] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6892.001428] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6892.013493] Lustre: Skipped 1 previous similar message [ 6892.015542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6893.779521] Lustre: server umount lustre-OST0000 complete [ 6895.233568] Lustre: server umount lustre-MDT0000 complete [ 6899.940202] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 6900.967451] Key type lgssc unregistered [ 6901.097426] LNet: 441306:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6901.100062] LNetError: 441306:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6901.106267] LNet: Removed LNI 192.168.204.144@tcp [ 6901.419570] Key type .llcrypt unregistered [ 6901.420755] Key type ._llcrypt unregistered [ 6909.526316] Key type ._llcrypt registered [ 6909.528393] Key type .llcrypt registered [ 6909.566493] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 6915.800826] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 6916.250906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6916.264587] alg: No test for adler32 (adler32-zlib) [ 6917.117331] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 6917.196553] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 6918.775139] Key type lgssc registered [ 6919.166847] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6931.056582] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 6936.122830] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6936.128424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6937.222050] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6937.235423] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6937.277678] Lustre: lustre-MDT0000: new disk, initializing [ 6937.303172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6937.310690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6938.628648] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6944.272147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6944.373400] Lustre: lustre-OST0000: new disk, initializing [ 6944.375456] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6944.378031] Lustre: Skipped 1 previous similar message [ 6944.380044] Lustre: 446312:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6944.402600] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6946.258665] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6947.069528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6947.073471] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6947.082594] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6949.997747] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6955.596670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6955.647215] Lustre: 445295:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6955.683155] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6955.721045] Lustre: lustre-MDT0001: new disk, initializing [ 6955.770797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6955.786087] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 6955.790296] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 6957.132458] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 6960.793373] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6960.876324] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6962.304156] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6963.402425] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 1 sec [ 6967.778200] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6967.778287] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6967.783273] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6967.786922] Lustre: Skipped 1 previous similar message [ 6968.597083] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.44@tcp (stopping) [ 6971.095974] Lustre: server umount lustre-MDT0001 complete [ 6978.017669] LustreError: 445305:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6978.022375] LustreError: 445305:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6983.138712] LustreError: 445301:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6983.143758] LustreError: 445301:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6988.256961] LustreError: 448622:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6988.257523] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6988.271508] Lustre: Skipped 1 previous similar message [ 6988.273201] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6988.273455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6989.519111] Lustre: server umount lustre-OST0000 complete [ 7005.151065] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7005.241485] Lustre: server umount lustre-MDT0000 complete [ 7009.775133] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 7010.801958] Key type lgssc unregistered [ 7010.927352] LNet: 450067:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7010.930598] LNetError: 450067:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7010.943341] LNet: Removed LNI 192.168.204.144@tcp [ 7011.236037] Key type .llcrypt unregistered [ 7011.237179] Key type ._llcrypt unregistered [ 7024.323931] Key type ._llcrypt registered [ 7024.325136] Key type .llcrypt registered [ 7024.375754] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 7024.495273] Key type .llcrypt unregistered [ 7024.497384] Key type ._llcrypt unregistered [ 7031.095112] Key type ._llcrypt registered [ 7031.096699] Key type .llcrypt registered [ 7031.132835] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 7031.418403] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7031.437443] alg: No test for adler32 (adler32-zlib) [ 7032.312901] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 7032.403974] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 7033.983196] Key type lgssc registered [ 7034.374382] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7043.043375] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7043.048491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7044.143443] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7044.154905] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7044.190852] Lustre: lustre-MDT0000: new disk, initializing [ 7044.213318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7044.219677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7045.623725] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7049.775175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7049.818231] Lustre: 453414:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7049.841051] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7049.843204] Lustre: Skipped 1 previous similar message [ 7049.879329] Lustre: lustre-MDT0001: new disk, initializing [ 7049.911654] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7049.922903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7049.925792] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7051.248844] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7055.298874] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7057.319542] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7059.589869] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7059.704837] Lustre: lustre-OST0000: new disk, initializing [ 7059.706480] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7059.708803] Lustre: 455380:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7059.740757] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7060.472931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7060.476761] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7060.511287] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7061.697808] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7065.507835] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7070.690190] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7070.691581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7070.694279] Lustre: Skipped 2 previous similar messages [ 7070.694481] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7070.697572] Lustre: Skipped 2 previous similar messages [ 7074.008857] Lustre: server umount lustre-MDT0000 complete [ 7075.225129] LustreError: 454208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781689312 with bad export cookie 14556240073989154119 [ 7075.228401] LustreError: 454208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7075.228919] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7075.346567] Lustre: server umount lustre-MDT0001 complete [ 7086.808600] Lustre: server umount lustre-OST0000 complete [ 7090.248416] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 05:42:06 (1781689326) [ 7092.732056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7092.905819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7094.232519] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7097.049358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7097.177057] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7098.574519] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7100.927708] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7102.740573] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7118.178444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7118.247221] Lustre: Found index 0 for lustre-OST0000, updating log [ 7118.308826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7120.179122] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7122.490651] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7138.789701] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 7146.975878] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7146.978242] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7146.982098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7148.001264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7148.003755] Lustre: Skipped 2 previous similar messages [ 7150.479845] Lustre: server umount lustre-OST0000 complete [ 7163.872025] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7163.876132] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7163.881008] Lustre: Skipped 1 previous similar message [ 7163.883860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7169.317412] Lustre: server umount lustre-MDT0000 complete [ 7169.504602] LustreError: 457404:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7169.509191] LustreError: 457404:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7170.635688] LustreError: 457385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781689407 with bad export cookie 14556240073989155498 [ 7170.639995] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7170.641032] LustreError: 457385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7170.769691] Lustre: server umount lustre-MDT0001 complete [ 7173.483126] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 05:43:29 (1781689409) [ 7181.919260] LNet: 1 local NIs in recovery (showing 1): 192.168.204.144@tcp [ 7754.208210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7754.366823] LustreError: 463827:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7754.391821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7755.784549] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7758.609582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7758.733058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7760.080660] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7762.480312] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7764.338249] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7779.618944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7779.727601] Lustre: Found index 0 for lustre-OST0000, updating log [ 7779.836397] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7781.776152] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7781.862119] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [ 7784.210943] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7785.953712] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7785.954758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7785.957037] Lustre: Skipped 1 previous similar message [ 7785.958862] Lustre: Skipped 2 previous similar messages [ 7786.977674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7786.980228] Lustre: Skipped 1 previous similar message [ 7788.001446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7791.052300] Lustre: server umount lustre-OST0000 complete [ 7793.312785] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7793.313937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7793.316734] Lustre: Skipped 1 previous similar message [ 7793.318938] Lustre: Skipped 1 previous similar message [ 7797.408819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7797.411594] Lustre: Skipped 8 previous similar messages [ 7798.426456] Lustre: server umount lustre-MDT0000 complete [ 7798.434233] LustreError: 463828:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7798.439211] LustreError: 463828:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7799.689337] LustreError: 464563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781690036 with bad export cookie 14556240073989156569 [ 7799.692421] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7799.695290] LustreError: 464563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7799.824317] Lustre: server umount lustre-MDT0001 complete [ 7808.607245] LNet: 1 local NIs in recovery (showing 1): 192.168.204.144@tcp [ 7817.051271] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 7820.094351] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 7831.964571] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 05:54:27 (1781690067) [ 7837.709274] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 7838.701977] Key type lgssc unregistered [ 7841.887820] LNet: 470473:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7841.891427] LNetError: 470473:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7910.501423] LNet: Removed LNI 192.168.204.144@tcp [ 7910.815021] Key type .llcrypt unregistered [ 7910.816456] Key type ._llcrypt unregistered [ 7917.510551] Key type ._llcrypt registered [ 7917.511986] Key type .llcrypt registered [ 7917.561185] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 7918.024481] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7918.058765] alg: No test for adler32 (adler32-zlib) [ 7918.920728] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 7919.007395] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 7920.583158] Key type lgssc registered [ 7920.941816] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7922.410426] LNet: Added LNI 192.168.204.144@tcp100 [8/256/0/180] [ 7922.707462] LNet: Added LNI 192.168.204.144@tcp101 [8/256/0/180] [ 7922.995085] LNet: Added LNI 192.168.204.144@tcp102 [8/256/0/180] [ 7923.289987] LNet: Added LNI 192.168.204.144@tcp103 [8/256/0/180] [ 7923.598620] LNet: Added LNI 192.168.204.144@tcp104 [8/256/0/180] [ 7923.893707] LNet: Added LNI 192.168.204.144@tcp105 [8/256/0/180] [ 7924.186360] LNet: Added LNI 192.168.204.144@tcp106 [8/256/0/180] [ 7924.475710] LNet: Added LNI 192.168.204.144@tcp107 [8/256/0/180] [ 7924.766358] LNet: Added LNI 192.168.204.144@tcp108 [8/256/0/180] [ 7925.060758] LNet: Added LNI 192.168.204.144@tcp109 [8/256/0/180] [ 7925.357889] LNet: Added LNI 192.168.204.144@tcp110 [8/256/0/180] [ 7925.665428] LNet: Added LNI 192.168.204.144@tcp111 [8/256/0/180] [ 7925.956727] LNet: Added LNI 192.168.204.144@tcp112 [8/256/0/180] [ 7926.249102] LNet: Added LNI 192.168.204.144@tcp113 [8/256/0/180] [ 7926.567465] LNet: Added LNI 192.168.204.144@tcp114 [8/256/0/180] [ 7926.861684] LNet: Added LNI 192.168.204.144@tcp115 [8/256/0/180] [ 7927.149909] LNet: Added LNI 192.168.204.144@tcp116 [8/256/0/180] [ 7927.446377] LNet: Added LNI 192.168.204.144@tcp117 [8/256/0/180] [ 7927.736345] LNet: Added LNI 192.168.204.144@tcp118 [8/256/0/180] [ 7928.027666] LNet: Added LNI 192.168.204.144@tcp119 [8/256/0/180] [ 7928.320337] LNet: Added LNI 192.168.204.144@tcp120 [8/256/0/180] [ 7928.623527] LNet: Added LNI 192.168.204.144@tcp121 [8/256/0/180] [ 7928.920700] LNet: Added LNI 192.168.204.144@tcp122 [8/256/0/180] [ 7929.216669] LNet: Added LNI 192.168.204.144@tcp123 [8/256/0/180] [ 7929.510387] LNet: Added LNI 192.168.204.144@tcp124 [8/256/0/180] [ 7929.799141] LNet: Added LNI 192.168.204.144@tcp125 [8/256/0/180] [ 7930.090808] LNet: Added LNI 192.168.204.144@tcp126 [8/256/0/180] [ 7930.380416] LNet: Added LNI 192.168.204.144@tcp127 [8/256/0/180] [ 7930.668442] LNet: Added LNI 192.168.204.144@tcp128 [8/256/0/180] [ 7930.958384] LNet: Added LNI 192.168.204.144@tcp129 [8/256/0/180] [ 7931.254164] LNet: Added LNI 192.168.204.144@tcp130 [8/256/0/180] [ 7931.542549] LNet: Added LNI 192.168.204.144@tcp131 [8/256/0/180] [ 7931.829579] LNet: Added LNI 192.168.204.144@tcp132 [8/256/0/180] [ 7932.118309] LNet: Added LNI 192.168.204.144@tcp133 [8/256/0/180] [ 7932.410707] LNet: Added LNI 192.168.204.144@tcp134 [8/256/0/180] [ 7932.699913] LNet: Added LNI 192.168.204.144@tcp135 [8/256/0/180] [ 7934.886568] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7934.891847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7935.943635] Lustre: 473469:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.144@tcp: can't add NID 192.168.204.144@tcp130: rc = -75 [ 7936.007092] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7936.017862] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7936.054850] Lustre: lustre-MDT0000: new disk, initializing [ 7936.086486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7936.092911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7937.425812] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7941.477468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7941.525702] Lustre: 473485:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7941.551158] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7941.552991] Lustre: Skipped 1 previous similar message [ 7941.587810] Lustre: lustre-MDT0001: new disk, initializing [ 7941.599902] Lustre: 474285:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.144@tcp: can't add NID 192.168.204.144@tcp130: rc = -75 [ 7941.602516] Lustre: 474285:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7941.640958] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7941.652388] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7941.655383] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7942.997492] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7943.284915] Lustre: 474582:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.144@tcp: can't add NID 192.168.204.144@tcp130: rc = -75 [ 7943.287947] Lustre: 474582:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7946.703399] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7948.512700] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7950.814502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7950.944380] Lustre: lustre-OST0000: new disk, initializing [ 7950.946088] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7950.948947] Lustre: 475460:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7950.990370] Lustre: 475465:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.144@tcp: can't add NID 192.168.204.144@tcp130: rc = -75 [ 7950.993011] Lustre: 475465:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [ 7951.012096] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7952.986796] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 7956.741765] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7959.160888] Lustre: server umount lustre-OST0000 complete [ 7960.559846] Lustre: 476602:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.144@tcp: can't add NID 192.168.204.144@tcp130: rc = -75 [ 7960.563205] Lustre: 476602:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7960.573360] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7960.574241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7960.577173] Lustre: Skipped 1 previous similar message [ 7965.664120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7965.666047] Lustre: Skipped 2 previous similar messages [ 7966.555504] Lustre: server umount lustre-MDT0000 complete [ 7967.797346] LustreError: 473477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781690204 with bad export cookie 6546625185504576481 [ 7967.800499] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7967.800989] LustreError: 473477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7967.918920] Lustre: server umount lustre-MDT0001 complete [ 7971.580219] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 7972.638076] Key type lgssc unregistered [ 7972.801639] LNet: 477362:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7972.804550] LNetError: 477362:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7972.806669] LNet: Removed LNI 192.168.204.144@tcp [ 7973.855129] LNet: Removed LNI 192.168.204.144@tcp100 [ 7973.856788] LNetError: 477362:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7973.860115] LNetError: 477362:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7973.863299] LNet: Removed LNI 192.168.204.144@tcp101 [ 7974.879167] LNet: Removed LNI 192.168.204.144@tcp102 [ 7974.880534] LNetError: 477362:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7974.882698] LNetError: 477362:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7974.885080] LNet: Removed LNI 192.168.204.144@tcp103 [ 7975.903153] LNet: Removed LNI 192.168.204.144@tcp104 [ 7976.927201] LNet: Removed LNI 192.168.204.144@tcp105 [ 7976.928698] LNetError: 477362:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7976.931363] LNetError: 477362:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 7976.934153] LNet: Removed LNI 192.168.204.144@tcp106 [ 7977.951157] LNet: Removed LNI 192.168.204.144@tcp107 [ 7978.975200] LNet: Removed LNI 192.168.204.144@tcp108 [ 7979.999156] LNet: Removed LNI 192.168.204.144@tcp109 [ 7981.023197] LNet: Removed LNI 192.168.204.144@tcp110 [ 7981.025568] LNetError: 477362:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7981.029586] LNetError: 477362:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 7981.033598] LNet: Removed LNI 192.168.204.144@tcp111 [ 7981.034889] LNet: Removed LNI 192.168.204.144@tcp112 [ 7981.036051] LNet: Removed LNI 192.168.204.144@tcp113 [ 7981.037235] LNet: Removed LNI 192.168.204.144@tcp114 [ 7981.038344] LNet: Removed LNI 192.168.204.144@tcp115 [ 7981.039576] LNet: Removed LNI 192.168.204.144@tcp116 [ 7981.040711] LNet: Removed LNI 192.168.204.144@tcp117 [ 7981.041803] LNet: Removed LNI 192.168.204.144@tcp118 [ 7981.043473] LNet: Removed LNI 192.168.204.144@tcp119 [ 7981.045541] LNet: Removed LNI 192.168.204.144@tcp120 [ 7981.047749] LNet: Removed LNI 192.168.204.144@tcp121 [ 7981.050089] LNet: Removed LNI 192.168.204.144@tcp122 [ 7981.052201] LNet: Removed LNI 192.168.204.144@tcp123 [ 7981.054227] LNet: Removed LNI 192.168.204.144@tcp124 [ 7981.056328] LNet: Removed LNI 192.168.204.144@tcp125 [ 7981.058372] LNet: Removed LNI 192.168.204.144@tcp126 [ 7981.060225] LNet: Removed LNI 192.168.204.144@tcp127 [ 7981.061850] LNet: Removed LNI 192.168.204.144@tcp128 [ 7981.063152] LNet: Removed LNI 192.168.204.144@tcp129 [ 7981.064347] LNet: Removed LNI 192.168.204.144@tcp130 [ 7981.065450] LNet: Removed LNI 192.168.204.144@tcp131 [ 7981.066740] LNet: Removed LNI 192.168.204.144@tcp132 [ 7981.067852] LNet: Removed LNI 192.168.204.144@tcp133 [ 7981.068980] LNet: Removed LNI 192.168.204.144@tcp134 [ 7981.076411] LNet: Removed LNI 192.168.204.144@tcp135 [ 7981.364356] Key type .llcrypt unregistered [ 7981.365482] Key type ._llcrypt unregistered [ 7988.794398] Key type ._llcrypt registered [ 7988.795709] Key type .llcrypt registered [ 7988.832533] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 7995.050811] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 7995.372409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7995.412858] alg: No test for adler32 (adler32-zlib) [ 7996.260388] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 7996.341983] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 7997.919178] Key type lgssc registered [ 7998.253928] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8009.254699] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 05:57:25 (1781690245) [ 8015.132064] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 8016.196180] Key type lgssc unregistered [ 8016.323392] LNet: 481420:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8016.326184] LNetError: 481420:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8016.339326] LNet: Removed LNI 192.168.204.144@tcp [ 8016.645390] Key type .llcrypt unregistered [ 8016.646774] Key type ._llcrypt unregistered [ 8023.192166] Key type ._llcrypt registered [ 8023.194148] Key type .llcrypt registered [ 8023.239891] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 8023.689814] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8023.739139] alg: No test for adler32 (adler32-zlib) [ 8024.600591] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 8024.693888] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 8026.279148] Key type lgssc registered [ 8026.685157] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8029.928763] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8029.937053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8031.048648] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8031.063914] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8031.102425] Lustre: lustre-MDT0000: new disk, initializing [ 8031.130367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8031.138319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8032.632185] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8036.874190] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8036.918486] Lustre: 482610:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8036.945931] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8036.947954] Lustre: Skipped 1 previous similar message [ 8036.983050] Lustre: lustre-MDT0001: new disk, initializing [ 8037.024845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8037.039567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8037.043925] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8038.420205] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8042.061342] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8043.778718] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8045.979198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8046.101825] Lustre: lustre-OST0000: new disk, initializing [ 8046.103776] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8046.106493] Lustre: 484577:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8046.137896] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8048.087896] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8051.704211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8051.707183] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8051.725475] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8051.746527] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8052.934598] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing lnet_if_list [ 8054.589288] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 8054.599795] LNet: Added LNI 192.168.204.144@tcp10 [8/256/0/180] [ 8056.459574] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 8056.473738] LNet: Added LNI 192.168.204.144@tcp11 [8/256/0/180] [ 8057.785339] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 8057.799596] LNet: Added LNI 192.168.204.144@tcp12 [8/256/0/180] [ 8059.116821] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 8059.127308] LNet: Added LNI 192.168.204.144@tcp13 [8/256/0/180] [ 8060.431236] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 8060.441196] LNet: Added LNI 192.168.204.144@tcp14 [8/256/0/180] [ 8061.872559] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 8061.883196] LNet: Added LNI 192.168.204.144@tcp15 [8/256/0/180] [ 8063.358217] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 8063.370280] LNet: Added LNI 192.168.204.144@tcp16 [8/256/0/180] [ 8064.819617] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 8064.831138] LNet: Added LNI 192.168.204.144@tcp17 [8/256/0/180] [ 8066.289461] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 8066.299772] LNet: Added LNI 192.168.204.144@tcp18 [8/256/0/180] [ 8067.828126] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 8067.842192] LNet: Added LNI 192.168.204.144@tcp19 [8/256/0/180] [ 8069.354224] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 8069.364219] LNet: Added LNI 192.168.204.144@tcp20 [8/256/0/180] [ 8071.013594] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 8071.025193] LNet: Added LNI 192.168.204.144@tcp21 [8/256/0/180] [ 8072.830225] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 8072.841475] LNet: Added LNI 192.168.204.144@tcp22 [8/256/0/180] [ 8074.654877] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 8074.669489] LNet: Added LNI 192.168.204.144@tcp23 [8/256/0/180] [ 8076.477352] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 8076.487982] LNet: Added LNI 192.168.204.144@tcp24 [8/256/0/180] [ 8078.267985] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 8078.278614] LNet: Added LNI 192.168.204.144@tcp25 [8/256/0/180] [ 8080.065408] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 8080.079905] LNet: Added LNI 192.168.204.144@tcp26 [8/256/0/180] [ 8081.888746] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 8081.901086] LNet: Added LNI 192.168.204.144@tcp27 [8/256/0/180] [ 8083.754481] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 8083.770270] LNet: Added LNI 192.168.204.144@tcp28 [8/256/0/180] [ 8085.723623] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 8085.733461] LNet: Added LNI 192.168.204.144@tcp29 [8/256/0/180] [ 8087.679037] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 8087.689835] LNet: Added LNI 192.168.204.144@tcp30 [8/256/0/180] [ 8089.618010] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 8089.628957] LNet: Added LNI 192.168.204.144@tcp31 [8/256/0/180] [ 8091.629230] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 8091.643439] LNet: Added LNI 192.168.204.144@tcp32 [8/256/0/180] [ 8093.756628] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 8093.767699] LNet: Added LNI 192.168.204.144@tcp33 [8/256/0/180] [ 8096.021150] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 8096.033985] LNet: Added LNI 192.168.204.144@tcp34 [8/256/0/180] [ 8098.221625] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 8098.231946] LNet: Added LNI 192.168.204.144@tcp35 [8/256/0/180] [ 8100.333067] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 8100.343705] LNet: Added LNI 192.168.204.144@tcp36 [8/256/0/180] [ 8102.552383] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 8102.563977] LNet: Added LNI 192.168.204.144@tcp37 [8/256/0/180] [ 8105.243829] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 8105.255775] LNet: Added LNI 192.168.204.144@tcp38 [8/256/0/180] [ 8107.696386] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 8107.705570] LNet: Added LNI 192.168.204.144@tcp39 [8/256/0/180] [ 8110.170210] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 8110.185201] LNet: Added LNI 192.168.204.144@tcp40 [8/256/0/180] [ 8112.553035] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 8112.562715] LNet: Added LNI 192.168.204.144@tcp41 [8/256/0/180] [ 8114.751527] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 8114.762072] LNet: Added LNI 192.168.204.144@tcp42 [8/256/0/180] [ 8117.098747] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 8117.109844] LNet: Added LNI 192.168.204.144@tcp43 [8/256/0/180] [ 8119.557741] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 8119.566387] LNet: Added LNI 192.168.204.144@tcp44 [8/256/0/180] [ 8122.004865] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 8122.015688] LNet: Added LNI 192.168.204.144@tcp45 [8/256/0/180] [ 8124.420368] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 8124.430781] LNet: Added LNI 192.168.204.144@tcp46 [8/256/0/180] [ 8126.976785] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 8126.987611] LNet: Added LNI 192.168.204.144@tcp47 [8/256/0/180] [ 8129.582966] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 8129.595431] LNet: Added LNI 192.168.204.144@tcp48 [8/256/0/180] [ 8132.468629] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 8132.480135] LNet: Added LNI 192.168.204.144@tcp49 [8/256/0/180] [ 8135.308467] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 8135.322424] LNet: Added LNI 192.168.204.144@tcp50 [8/256/0/180] [ 8143.841540] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8143.843114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8143.846298] Lustre: Skipped 2 previous similar messages [ 8143.847946] Lustre: Skipped 1 previous similar message [ 8148.893411] Lustre: server umount lustre-MDT0000 complete [ 8150.123526] LustreError: 483406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781690386 with bad export cookie 2786436340964060484 [ 8150.127957] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8150.129471] LustreError: 483406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8150.246311] Lustre: server umount lustre-MDT0001 complete [ 8157.480703] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 8160.344475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8169.439195] Lustre: 481883:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781690390/real 1781690390] req@0000000079815301 x1868237647199488/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781690406 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8169.445845] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8170.463162] Lustre: 481884:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781690391/real 1781690391] req@000000000e06e818 x1868237647199616/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781690407 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8174.559146] Lustre: 481885:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781690395/real 1781690395] req@00000000cd7ec7b3 x1868237647199744/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781690411 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8174.560910] LustreError: 481881:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000001288fb82 x1868237647200512/t0(0) o250->MGC192.168.204.144@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8174.725521] LustreError: 484570:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8174.731815] LustreError: 484570:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 8174.750920] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8176.109949] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8178.786515] LustreError: 484571:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8179.101992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8179.242340] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8180.213128] Lustre: 481883:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781690401/real 1781690401] req@0000000031fc89f0 x1868237647200000/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781690417 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8180.260582] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8180.723865] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8183.145500] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8185.038445] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8186.335114] Lustre: 481882:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781690406/real 1781690406] req@000000004123061e x1868237647200128/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781690422 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8194.528210] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8194.530323] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8194.533936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8194.536744] Lustre: Skipped 1 previous similar message [ 8195.554239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8195.555954] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8198.483915] Lustre: server umount lustre-OST0000 complete [ 8200.673868] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8200.675224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8200.678578] Lustre: Skipped 2 previous similar messages [ 8200.681302] Lustre: Skipped 1 previous similar message [ 8205.792935] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8205.796111] Lustre: Skipped 1 previous similar message [ 8205.927507] Lustre: server umount lustre-MDT0000 complete [ 8207.332512] LustreError: 482603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781690444 with bad export cookie 2786436340964062500 [ 8207.336716] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8207.337841] LustreError: 482603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8207.469310] Lustre: server umount lustre-MDT0001 complete [ 8211.176809] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 8212.299103] Key type lgssc unregistered [ 8212.488159] LNet: 511537:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8212.495685] LNetError: 511537:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8212.499284] LNet: Removed LNI 192.168.204.144@tcp [ 8212.501163] LNet: Removed LNI 192.168.204.144@tcp10 [ 8212.503087] LNet: Removed LNI 192.168.204.144@tcp11 [ 8212.504889] LNet: Removed LNI 192.168.204.144@tcp12 [ 8212.506708] LNet: Removed LNI 192.168.204.144@tcp13 [ 8212.508597] LNet: Removed LNI 192.168.204.144@tcp14 [ 8212.510668] LNet: Removed LNI 192.168.204.144@tcp15 [ 8212.512554] LNet: Removed LNI 192.168.204.144@tcp16 [ 8212.514127] LNet: Removed LNI 192.168.204.144@tcp17 [ 8212.515772] LNet: Removed LNI 192.168.204.144@tcp18 [ 8212.517390] LNet: Removed LNI 192.168.204.144@tcp19 [ 8212.518884] LNet: Removed LNI 192.168.204.144@tcp20 [ 8212.520547] LNet: Removed LNI 192.168.204.144@tcp21 [ 8212.522106] LNet: Removed LNI 192.168.204.144@tcp22 [ 8212.523700] LNet: Removed LNI 192.168.204.144@tcp23 [ 8212.525113] LNet: Removed LNI 192.168.204.144@tcp24 [ 8212.526623] LNet: Removed LNI 192.168.204.144@tcp25 [ 8212.527960] LNet: Removed LNI 192.168.204.144@tcp26 [ 8212.529659] LNet: Removed LNI 192.168.204.144@tcp27 [ 8212.531273] LNet: Removed LNI 192.168.204.144@tcp28 [ 8212.532676] LNet: Removed LNI 192.168.204.144@tcp29 [ 8212.534299] LNet: Removed LNI 192.168.204.144@tcp30 [ 8212.535873] LNet: Removed LNI 192.168.204.144@tcp31 [ 8212.537123] LNet: Removed LNI 192.168.204.144@tcp32 [ 8212.538682] LNet: Removed LNI 192.168.204.144@tcp33 [ 8212.540488] LNet: Removed LNI 192.168.204.144@tcp34 [ 8212.542199] LNet: Removed LNI 192.168.204.144@tcp35 [ 8212.543874] LNet: Removed LNI 192.168.204.144@tcp36 [ 8212.545370] LNet: Removed LNI 192.168.204.144@tcp37 [ 8212.546942] LNet: Removed LNI 192.168.204.144@tcp38 [ 8212.548145] LNet: Removed LNI 192.168.204.144@tcp39 [ 8212.549374] LNet: Removed LNI 192.168.204.144@tcp40 [ 8212.550582] LNet: Removed LNI 192.168.204.144@tcp41 [ 8212.551806] LNet: Removed LNI 192.168.204.144@tcp42 [ 8212.553120] LNet: Removed LNI 192.168.204.144@tcp43 [ 8212.555245] LNet: Removed LNI 192.168.204.144@tcp44 [ 8212.557659] LNet: Removed LNI 192.168.204.144@tcp45 [ 8212.558756] LNet: Removed LNI 192.168.204.144@tcp46 [ 8212.559803] LNet: Removed LNI 192.168.204.144@tcp47 [ 8212.561108] LNet: Removed LNI 192.168.204.144@tcp48 [ 8212.562624] LNet: Removed LNI 192.168.204.144@tcp49 [ 8212.571407] LNet: Removed LNI 192.168.204.144@tcp50 [ 8212.922588] Key type .llcrypt unregistered [ 8212.924362] Key type ._llcrypt unregistered [ 8219.881448] Key type ._llcrypt registered [ 8219.882798] Key type .llcrypt registered [ 8219.926743] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 8225.836377] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 8226.217283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8226.228105] alg: No test for adler32 (adler32-zlib) [ 8227.128306] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 8227.222075] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 8228.807203] Key type lgssc registered [ 8229.157532] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8240.417871] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8240.425059] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8241.528721] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8241.539732] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8241.577406] Lustre: lustre-MDT0000: new disk, initializing [ 8241.604421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8241.611417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8242.970800] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8246.993469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8247.032710] Lustre: 514717:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8247.070350] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8247.072234] Lustre: Skipped 1 previous similar message [ 8247.117732] Lustre: lustre-MDT0001: new disk, initializing [ 8247.152192] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8247.166204] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8247.169342] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8248.526588] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8252.171609] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8253.888248] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8256.353668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8256.438734] Lustre: *** cfs_fail_loc=90f, val=0*** [ 8256.475244] Lustre: lustre-OST0000: new disk, initializing [ 8256.477983] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8256.482059] Lustre: 516733:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8256.515299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8256.838713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8256.843812] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8256.893076] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8258.582635] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8262.437180] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8267.233620] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8267.235399] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8267.236189] LustreError: Skipped 1 previous similar message [ 8267.242712] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8267.246167] Lustre: Skipped 1 previous similar message [ 8270.803157] Lustre: server umount lustre-OST0000 complete [ 8272.352366] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8272.352940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8272.355620] Lustre: Skipped 2 previous similar messages [ 8272.360283] Lustre: Skipped 1 previous similar message [ 8277.472323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8277.475661] Lustre: Skipped 1 previous similar message [ 8278.171008] Lustre: server umount lustre-MDT0000 complete [ 8279.314338] LustreError: 514710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781690516 with bad export cookie 11766394809133440798 [ 8279.317490] LustreError: 514710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8279.317667] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8279.433587] Lustre: server umount lustre-MDT0001 complete [ 8283.079011] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 8284.161455] Key type lgssc unregistered [ 8284.281414] LNet: 518636:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8284.283716] LNetError: 518636:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8284.295276] LNet: Removed LNI 192.168.204.144@tcp [ 8284.578813] Key type .llcrypt unregistered [ 8284.580033] Key type ._llcrypt unregistered [ 8291.658997] Key type ._llcrypt registered [ 8291.660098] Key type .llcrypt registered [ 8291.704559] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 8292.084579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8292.127072] alg: No test for adler32 (adler32-zlib) [ 8292.984557] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 8293.073267] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 8294.663192] Key type lgssc registered [ 8295.050311] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8297.635453] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8297.642677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8298.852428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8300.228634] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8303.100360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8303.232396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8304.559131] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8306.901767] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8308.717844] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8310.982529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8311.107615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8313.072571] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8315.434890] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8329.457332] loop0: detected capacity change from 0 to 204800000 [ 8329.532536] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8329.536287] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8329.541175] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8331.519833] loop0: detected capacity change from 0 to 204800000 [ 8331.602347] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8331.606864] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8331.611758] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8333.397687] loop0: detected capacity change from 0 to 204800000 [ 8335.372043] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8335.432165] Lustre: fs15246-OST0000: new disk, initializing [ 8335.434260] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 8335.438594] Lustre: 523494:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8335.463369] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8337.524849] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8341.389826] loop1: detected capacity change from 0 to 204800000 [ 8343.233537] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8343.270888] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 8343.294507] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 8343.334826] Lustre: fs15246-MDT0000: new disk, initializing [ 8343.389256] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8343.399105] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8343.450337] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 8344.480221] Lustre: 524616:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@000000000c1759e8 x1868237928168192/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1781690597 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 8344.891982] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8353.567322] LustreError: 524616:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 8353.573470] LustreError: 524616:0:(fid_request.c:259:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [ 8353.579134] LustreError: 524616:0:(osp_precreate.c:462:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [ 8356.325342] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8356.332261] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 8358.800889] Lustre: server umount fs15246-MDT0000 complete [ 8361.444501] LustreError: 519880:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8361.451764] LustreError: 519880:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8361.934812] Lustre: server umount fs15246-OST0000 complete [ 8366.562599] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8366.563259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8366.567384] Lustre: Skipped 1 previous similar message [ 8371.418888] Lustre: server umount lustre-OST0000 complete [ 8375.264222] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8375.267832] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8375.274390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8375.276267] Lustre: Skipped 1 previous similar message [ 8381.920263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8381.923225] Lustre: Skipped 3 previous similar messages [ 8387.039183] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8387.040821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8387.043524] Lustre: Skipped 1 previous similar message [ 8387.099219] Lustre: server umount lustre-MDT0000 complete [ 8388.340354] LustreError: 519864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781690625 with bad export cookie 2331288376303217381 [ 8388.343453] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8388.343666] LustreError: 519864:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8388.461991] Lustre: server umount lustre-MDT0001 complete [ 8391.979830] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 8392.988482] Key type lgssc unregistered [ 8393.117316] LNet: 527710:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8393.119693] LNetError: 527710:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8393.133416] LNet: Removed LNI 192.168.204.144@tcp [ 8393.433090] Key type .llcrypt unregistered [ 8393.434129] Key type ._llcrypt unregistered [ 8400.541475] Key type ._llcrypt registered [ 8400.543167] Key type .llcrypt registered [ 8400.590321] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 8401.027756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8401.095725] alg: No test for adler32 (adler32-zlib) [ 8401.965590] Lustre: Lustre: Build Version: 2.17.54_40_g42483b8 [ 8402.070092] LNet: Added LNI 192.168.204.144@tcp [8/256/0/180] [ 8403.647204] Key type lgssc registered [ 8404.033141] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8416.521394] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 8419.626658] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 8431.152379] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8431.157702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8432.252968] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8432.263556] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8432.302030] Lustre: lustre-MDT0000: new disk, initializing [ 8432.326968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8432.334402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8433.603562] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8437.686481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8437.727514] Lustre: 532146:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8437.750142] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8437.751711] Lustre: Skipped 1 previous similar message [ 8437.795447] Lustre: lustre-MDT0001: new disk, initializing [ 8437.828944] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8437.841402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8437.844660] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8439.104128] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8442.812254] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8444.487954] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8446.654189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8446.765180] Lustre: lustre-OST0000: new disk, initializing [ 8446.766703] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8446.768954] Lustre: 534111:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8446.796962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8447.139110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8447.144342] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8447.183856] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8448.832988] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_default_debug -1 all [ 8452.612284] Lustre: DEBUG MARKER: oleg444-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8454.038538] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8454.102171] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8455.501418] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8455.579220] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8457.698348] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8457.698391] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8457.700307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8457.702992] LustreError: Skipped 1 previous similar message [ 8457.707665] Lustre: Skipped 1 previous similar message [ 8462.307167] Lustre: server umount lustre-OST0000 complete [ 8467.936787] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8467.937446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8467.942400] Lustre: Skipped 1 previous similar message [ 8467.943962] Lustre: Skipped 1 previous similar message [ 8469.729888] Lustre: server umount lustre-MDT0000 complete [ 8470.932206] LustreError: 532137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781690707 with bad export cookie 9726832818896974717 [ 8470.934725] LustreError: MGC192.168.204.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8470.935326] LustreError: 532137:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8471.055643] Lustre: server umount lustre-MDT0001 complete [ 8479.021910] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing set_hostid [ 8482.039179] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing load_modules_local [ 8492.112701] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8167 sec ========== 06:05:28 (1781690728) [ 8492.648666] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 06:05:28 (1781690728) === [ 8493.748623] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 06:05:29 (1781690729) === [ 8506.405724] Lustre: DEBUG MARKER: oleg444-server.virtnet: executing unload_modules_local [ 8507.397837] Key type lgssc unregistered [ 8507.531352] LNet: 540738:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8507.534072] LNetError: 540738:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8507.543300] LNet: Removed LNI 192.168.204.144@tcp [ 8507.817615] Key type .llcrypt unregistered [ 8507.818645] Key type ._llcrypt unregistered