[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 481062531 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003242] x2apic enabled [ 0.004011] Switched APIC routing to physical x2apic. [ 0.005017] kvm-guest: setup PV IPIs [ 0.009000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010013] pid_max: default: 32768 minimum: 301 [ 0.011139] LSM: Security Framework initializing [ 0.012064] Yama: becoming mindful. [ 0.013061] SELinux: Initializing. [ 0.014054] *** VALIDATE selinux *** [ 0.023375] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028446] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029176] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031098] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032124] *** VALIDATE tmpfs *** [ 0.033501] *** VALIDATE proc *** [ 0.035172] *** VALIDATE cgroup *** [ 0.036013] *** VALIDATE cgroup2 *** [ 0.038184] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039164] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040013] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041031] Spectre V2 : User space: Vulnerable [ 0.042011] Speculative Store Bypass: Vulnerable [ 0.044755] debug: unmapping init [mem 0xffffffffb6059000-0xffffffffb6060fff] [ 0.046849] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047718] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048022] ... version: 2 [ 0.049016] ... bit width: 48 [ 0.050014] ... generic registers: 4 [ 0.051014] ... value mask: 0000ffffffffffff [ 0.052015] ... max period: 00007fffffffffff [ 0.053013] ... fixed-purpose events: 3 [ 0.054013] ... event mask: 000000070000000f [ 0.055312] rcu: Hierarchical SRCU implementation. [ 0.057542] smp: Bringing up secondary CPUs ... [ 0.058727] x86: Booting SMP configuration: [ 0.059028] .... node #0, CPUs: #1 #2 #3 [ 0.062307] smp: Brought up 1 node, 4 CPUs [ 0.064014] smpboot: Max logical packages: 1 [ 0.065021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.143555] node 0 deferred pages initialised in 76ms [ 0.147095] devtmpfs: initialized [ 0.148313] x86/mm: Memory block size: 128MB [ 0.151840] gcov: version magic: 0x41383552 [ 0.155294] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.158124] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.160356] pinctrl core: initialized pinctrl subsystem [ 0.162192] [ 0.162693] ************************************************************* [ 0.165011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.166011] ** ** [ 0.168012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.169010] ** ** [ 0.171010] ** This means that this kernel is built to expose internal ** [ 0.172008] ** IOMMU data structures, which may compromise security on ** [ 0.174009] ** your system. ** [ 0.175008] ** ** [ 0.177009] ** If you see this message and you are not debugging the ** [ 0.178009] ** kernel, report this immediately to your vendor! ** [ 0.180010] ** ** [ 0.181007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.183014] ************************************************************* [ 0.185662] NET: Registered protocol family 16 [ 0.187448] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.189056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.191056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.194128] cpuidle: using governor menu [ 0.195943] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.198470] PCI: Using configuration type 1 for base access [ 0.201148] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.211124] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.215029] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.218145] cryptd: max_cpu_qlen set to 1000 [ 0.220191] ACPI: Added _OSI(Module Device) [ 0.222021] ACPI: Added _OSI(Processor Device) [ 0.223015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.225022] ACPI: Added _OSI(Processor Aggregator Device) [ 0.230799] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.237363] ACPI: Interpreter enabled [ 0.239062] ACPI: PM: (supports S0 S3 S4 S5) [ 0.240012] ACPI: Using IOAPIC for interrupt routing [ 0.241103] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.245107] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.255870] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.257037] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.259027] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.263100] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.268482] acpiphp: Slot [2] registered [ 0.270116] acpiphp: Slot [5] registered [ 0.271133] acpiphp: Slot [6] registered [ 0.273170] acpiphp: Slot [7] registered [ 0.275154] acpiphp: Slot [8] registered [ 0.276104] acpiphp: Slot [9] registered [ 0.278103] acpiphp: Slot [10] registered [ 0.279117] acpiphp: Slot [3] registered [ 0.281151] acpiphp: Slot [4] registered [ 0.282107] acpiphp: Slot [11] registered [ 0.284151] acpiphp: Slot [12] registered [ 0.285266] acpiphp: Slot [13] registered [ 0.287108] acpiphp: Slot [14] registered [ 0.288113] acpiphp: Slot [15] registered [ 0.290096] acpiphp: Slot [16] registered [ 0.291084] acpiphp: Slot [17] registered [ 0.292000] acpiphp: Slot [18] registered [ 0.292000] acpiphp: Slot [19] registered [ 0.293091] acpiphp: Slot [20] registered [ 0.295098] acpiphp: Slot [21] registered [ 0.296102] acpiphp: Slot [22] registered [ 0.298108] acpiphp: Slot [23] registered [ 0.299109] acpiphp: Slot [24] registered [ 0.301148] acpiphp: Slot [25] registered [ 0.302099] acpiphp: Slot [26] registered [ 0.304106] acpiphp: Slot [27] registered [ 0.305094] acpiphp: Slot [28] registered [ 0.307103] acpiphp: Slot [29] registered [ 0.308069] acpiphp: Slot [30] registered [ 0.309102] acpiphp: Slot [31] registered [ 0.311063] PCI host bridge to bus 0000:00 [ 0.312022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.315025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.317020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.320037] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.323026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.326029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.328193] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.331044] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.335417] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.346015] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.349357] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.352018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.355018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.357015] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.362582] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.364742] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.368035] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.371043] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.376027] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.390019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.396970] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.401000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.407019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.413017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.431016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.439731] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.447017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.454014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.470017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.479624] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.488017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.496017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.512022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.521615] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.529018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.534022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.548019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.559000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.569018] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.576015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.592019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.601566] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.607017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.612015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.629025] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.642758] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.645484] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.649595] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.652560] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.655270] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.662117] iommu: Default domain type: Passthrough [ 0.663442] SCSI subsystem initialized [ 0.665142] ACPI: bus type USB registered [ 0.666123] usbcore: registered new interface driver usbfs [ 0.668161] usbcore: registered new interface driver hub [ 0.671139] usbcore: registered new device driver usb [ 0.673243] pps_core: LinuxPPS API ver. 1 registered [ 0.675014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.679067] PTP clock support registered [ 0.681094] EDAC MC: Ver: 3.0.0 [ 0.683165] PCI: Using ACPI for IRQ routing [ 0.685010] NetLabel: Initializing [ 0.686011] NetLabel: domain hash size = 128 [ 0.687012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.688095] NetLabel: unlabeled traffic allowed by default [ 0.690128] vgaarb: loaded [ 0.692270] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.693017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.702346] clocksource: Switched to clocksource kvm-clock [ 0.810305] VFS: Disk quotas dquot_6.6.0 [ 0.811911] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.814335] *** VALIDATE ramfs *** [ 0.815270] *** VALIDATE hugetlbfs *** [ 0.816866] pnp: PnP ACPI init [ 0.819449] pnp: PnP ACPI: found 6 devices [ 0.846250] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.851354] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.855635] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.857719] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.860030] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.862545] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.866830] NET: Registered protocol family 2 [ 0.869415] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.874247] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.878297] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.883827] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.887220] TCP: Hash tables configured (established 65536 bind 65536) [ 0.889807] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.892851] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.896198] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.899370] NET: Registered protocol family 1 [ 0.902381] RPC: Registered named UNIX socket transport module. [ 0.904396] RPC: Registered udp transport module. [ 0.906459] RPC: Registered tcp transport module. [ 0.908247] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.911034] NET: Registered protocol family 44 [ 0.912431] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.913942] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.916139] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.918342] PCI: CLS 0 bytes, default 64 [ 0.919682] Unpacking initramfs... [ 2.374781] debug: unmapping init [mem 0xffff9daafcc54000-0xffff9daafffbffff] [ 2.381973] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.384329] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.389213] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.898233] Initialise system trusted keyrings [ 2.902084] Key type blacklist registered [ 2.904019] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.912996] zbud: loaded [ 2.916103] *** VALIDATE nfs *** [ 2.917371] *** VALIDATE nfs4 *** [ 2.918982] pstore: using deflate compression [ 2.923303] Platform Keyring initialized [ 3.036141] NET: Registered protocol family 38 [ 3.038424] Key type asymmetric registered [ 3.040162] Asymmetric key parser 'x509' registered [ 3.042381] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.046153] io scheduler mq-deadline registered [ 3.048019] io scheduler kyber registered [ 3.049684] io scheduler bfq registered [ 3.051633] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.054882] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.058059] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.061445] ACPI: Power Button [PWRF] [ 3.066964] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.073981] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.089868] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.101578] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.127437] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.156563] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.186580] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.191958] Non-volatile memory driver v1.3 [ 3.194118] Linux agpgart interface v0.103 [ 3.230648] virtio_blk virtio1: [vda] 146016 512-byte logical blocks (74.8 MB/71.3 MiB) [ 3.234054] vda: detected capacity change from 0 to 74760192 [ 3.248441] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.251817] vdb: detected capacity change from 0 to 1073741824 [ 3.268403] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.271906] vdc: detected capacity change from 0 to 2621440000 [ 3.288783] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.292287] vdd: detected capacity change from 0 to 2621440000 [ 3.307287] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.310275] vde: detected capacity change from 0 to 4294967296 [ 3.327396] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.331025] vdf: detected capacity change from 0 to 4294967296 [ 3.339724] libphy: Fixed MDIO Bus: probed [ 3.354728] usbcore: registered new interface driver usbserial_generic [ 3.358074] usbserial: USB Serial support registered for generic [ 3.360022] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.363826] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.365673] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.368237] mousedev: PS/2 mouse device common for all mice [ 3.371910] rtc_cmos 00:05: RTC can wake from S4 [ 3.375391] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.376705] rtc_cmos 00:05: registered as rtc0 [ 3.384889] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.388863] intel_pstate: CPU model not supported [ 3.392239] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.394147] hid: raw HID events driver (C) Jiri Kosina [ 3.398569] usbcore: registered new interface driver usbhid [ 3.401104] usbhid: USB HID core driver [ 3.401406] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.402902] drop_monitor: Initializing network drop monitor service [ 3.409236] Initializing XFRM netlink socket [ 3.411849] NET: Registered protocol family 10 [ 3.415428] Segment Routing with IPv6 [ 3.416880] NET: Registered protocol family 17 [ 3.419440] mpls_gso: MPLS GSO support [ 3.426235] RAS: Correctable Errors collector initialized. [ 3.428802] AVX version of gcm_enc/dec engaged. [ 3.430958] AES CTR mode by8 optimization enabled [ 3.519543] sched_clock: Marking stable (3519515714, 0)->(4455048151, -935532437) [ 3.523570] registered taskstats version 1 [ 3.525772] Loading compiled-in X.509 certificates [ 3.528026] zswap: loaded using pool lzo/zbud [ 3.554465] Key type big_key registered [ 3.568534] Key type encrypted registered [ 3.570769] ima: No TPM chip found, activating TPM-bypass! [ 3.573258] ima: Allocated hash algorithm: sha1 [ 3.575318] ima: No architecture policies found [ 3.577622] evm: Initialising EVM extended attributes: [ 3.580535] evm: security.selinux [ 3.581562] evm: security.ima [ 3.582430] evm: security.capability [ 3.583472] evm: HMAC attrs: 0x1 [ 3.587456] rtc_cmos 00:05: setting system clock to 2026-08-31 05:18:45 UTC (1788153525) [ 3.593791] debug: unmapping init [mem 0xffffffffb7003000-0xffffffffb71fffff] [ 3.596274] debug: unmapping init [mem 0xffffffffb5d82000-0xffffffffb6058fff] [ 3.604117] Write protecting the kernel read-only data: 28672k [ 3.607635] debug: unmapping init [mem 0xffffffffb4403000-0xffffffffb45fffff] [ 3.610811] debug: unmapping init [mem 0xffffffffb4d14000-0xffffffffb4dfffff] [ 3.648367] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.657807] systemd[1]: Detected virtualization kvm. [ 3.659969] systemd[1]: Detected architecture x86-64. [ 3.661348] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.691030] systemd[1]: No hostname configured. [ 3.692547] systemd[1]: Set hostname to . [ 3.694495] random: systemd: uninitialized urandom read (16 bytes read) [ 3.696757] systemd[1]: Initializing machine ID from random generator. [ 3.763858] random: ln: uninitialized urandom read (6 bytes read) [ 3.858404] random: systemd: uninitialized urandom read (16 bytes read) [ 3.861693] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.868821] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ 3.873554] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Reached target Timers. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.553422] device-mapper: uevent: version 1.0.3 [ 4.555960] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.298926] virtio_net virtio0 ens2: renamed from eth0 [ 5.327607] random: fast init done [ 5.354157] scsi host0: ata_piix [ 5.369471] scsi host1: ata_piix [ 5.380306] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.383542] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.851039] dracut-initqueue[588]: RTNETLINK answers: File exists [ 10.091284] random: crng init done [ 10.093179] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.599525] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.863479] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.183798] SELinux: Disabled at runtime. [ 12.249122] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.259134] systemd[1]: Detected virtualization kvm. [ 12.262208] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.844322] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.848107] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.853703] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.857861] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.861234] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.868214] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.877932] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... [[ 12.998919] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS  OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... Starting udev Coldplug all Devices... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.413261] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.767344] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.790889] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.016443] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.027150] EDAC sbridge: Ver: 1.1.2 [ 15.941216] Key type dns_resolver registered [ 16.296961] NFS: Registering the id_resolver key type [ 16.299249] Key type id_resolver registered [ 16.300968] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Login Service... Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg439-server login: [ 60.243404] libcfs: loading out-of-tree module taints kernel. [ 60.289757] Key type ._llcrypt registered [ 60.294644] Key type .llcrypt registered [ 60.418499] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_hostid [ 78.795865] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [ 80.371289] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 80.382553] alg: No test for adler32 (adler32-zlib) [ 82.132460] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 83.241119] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [ 84.999176] Key type lgssc registered [ 87.498698] Lustre: Echo OBD driver; http://www.lustre.org/ [ 107.078661] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 117.131007] hrtimer: interrupt took 3998957 ns [ 148.808364] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [ 161.523562] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 161.561860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 162.817701] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 162.865379] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 162.958823] Lustre: lustre-MDT0000: new disk, initializing [ 163.039129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 163.057226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 167.070393] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 179.708927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 179.817701] Lustre: 6510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 179.868699] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 179.873771] Lustre: Skipped 1 previous similar message [ 179.962489] Lustre: lustre-MDT0001: new disk, initializing [ 180.052370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 180.088575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 180.114876] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 184.746546] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 189.472097] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 198.676723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 198.923553] Lustre: lustre-OST0000: new disk, initializing [ 198.941030] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 198.953951] Lustre: 8448:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 199.033650] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 205.822963] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 207.406966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 207.424900] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 207.522613] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 219.723226] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 219.862720] Lustre: lustre-OST0001: new disk, initializing [ 219.868372] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 219.872382] Lustre: 9519:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 219.939675] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 226.319816] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 229.424311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 229.456577] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 229.514594] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 238.400460] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 244.577425] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 249.971481] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing check_logdir /tmp/testlogs/ [ 254.085879] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing yml_node [ 258.145169] Lustre: DEBUG MARKER: Client: 2.17.55.27 [ 260.329209] Lustre: DEBUG MARKER: MDS: 2.17.55.27 [ 262.663855] Lustre: DEBUG MARKER: OSS: 2.17.55.27 [ 264.123735] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 31 01:23:04 EDT 2026 [ 278.216096] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 279.642952] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 295.904144] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 295.910142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 295.923527] Lustre: Skipped 2 previous similar messages [ 295.935147] Lustre: Skipped 3 previous similar messages [ 301.025116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 301.041057] Lustre: Skipped 3 previous similar messages [ 301.446649] Lustre: server umount lustre-MDT0000 complete [ 305.012804] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153826 with bad export cookie 1117314957788165975 [ 305.016059] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 305.036324] LustreError: 6501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 306.143830] LustreError: 6515:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 306.144323] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 306.144865] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 306.161996] LustreError: 6515:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 306.221932] Lustre: Skipped 2 previous similar messages [ 310.176770] LustreError: 6516:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 310.176917] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 310.192859] LustreError: 6516:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 310.204550] Lustre: Skipped 2 previous similar messages [ 311.313911] Lustre: server umount lustre-MDT0001 complete [ 314.752388] Lustre: server umount lustre-OST0000 complete [ 317.919595] Lustre: server umount lustre-OST0001 complete [ 322.752346] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_hostid [ 330.882669] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [ 364.558616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 364.754651] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 364.779745] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 364.892329] Lustre: lustre-MDT0000: new disk, initializing [ 365.022430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 365.046647] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 368.918706] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 380.275687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 380.375161] Lustre: 15069:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 380.431244] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 380.438382] Lustre: Skipped 1 previous similar message [ 380.559844] Lustre: lustre-MDT0001: new disk, initializing [ 380.748367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 380.826512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 380.843063] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 386.247517] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 395.892358] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 400.900679] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 407.895238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 408.140299] Lustre: lustre-OST0000: new disk, initializing [ 408.144891] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 408.149514] Lustre: 17038:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 408.229960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 409.284976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 409.295991] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 409.492466] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 414.616417] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 423.330827] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 427.452112] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 427.623937] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 432.007253] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 432.198292] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 435.168774] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 435.176091] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 435.177649] LustreError: Skipped 1 previous similar message [ 435.182642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 435.191391] Lustre: Skipped 1 previous similar message [ 435.211829] Lustre: Skipped 1 previous similar message [ 440.065719] Lustre: server umount lustre-OST0000 complete [ 445.415080] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 445.449876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 445.455445] Lustre: Skipped 1 previous similar message [ 450.086458] Lustre: server umount lustre-MDT0000 complete [ 450.530723] LustreError: 15082:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 450.568082] LustreError: 15082:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 454.038308] LustreError: 15062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153975 with bad export cookie 1117314957788167508 [ 454.040421] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 454.048588] LustreError: 15062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 454.441409] Lustre: server umount lustre-MDT0001 complete [ 458.536700] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 01:26:19 (1788153979) [ 460.919462] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 462.533811] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 01:26:23 (1788153983) [ 471.218181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 471.821833] LustreError: 19449:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 471.926619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 476.976319] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 477.160632] LustreError: 19450:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 486.285974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 486.378015] LustreError: 19450:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 486.408349] LustreError: 19450:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 491.591582] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 499.688853] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 505.364493] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 512.507172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 519.480918] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 527.850994] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 541.665516] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 541.671892] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 541.685571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 541.729961] Lustre: Skipped 2 previous similar messages [ 545.988622] Lustre: server umount lustre-OST0000 complete [ 551.932517] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 556.771136] Lustre: server umount lustre-MDT0000 complete [ 557.032615] LustreError: 19450:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 560.909988] LustreError: 19430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154082 with bad export cookie 1117314957788168390 [ 560.922372] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 560.935272] LustreError: 19430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 561.243217] Lustre: server umount lustre-MDT0001 complete [ 573.039491] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [ 576.318398] Key type lgssc unregistered [ 576.551648] LNet: 22982:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 576.560980] LNetError: 22982:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 576.577229] LNet: Removed LNI 192.168.204.139@tcp [ 577.641070] Key type .llcrypt unregistered [ 577.643870] Key type ._llcrypt unregistered [ 601.010765] Key type ._llcrypt registered [ 601.013627] Key type .llcrypt registered [ 601.147641] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [ 602.083854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 602.208084] alg: No test for adler32 (adler32-zlib) [ 603.298917] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 603.515148] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [ 605.175164] Key type lgssc registered [ 606.305465] Lustre: Echo OBD driver; http://www.lustre.org/ [ 614.138319] loop: module loaded [ 614.201532] loop0: detected capacity change from 0 to 204800000 [ 614.481254] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 614.488602] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 614.830485] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 620.987933] loop0: detected capacity change from 0 to 204800000 [ 621.168124] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 621.183755] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 621.203220] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 626.546276] loop0: detected capacity change from 0 to 204800000 [ 632.003342] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 632.028622] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 633.288404] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 633.355985] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 633.456837] Lustre: lustre-MDT0000: new disk, initializing [ 633.553471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 633.583737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 637.826924] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 646.974464] loop1: detected capacity change from 0 to 204800000 [ 652.869883] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 653.297415] Lustre: lustre-OST07c6: new disk, initializing [ 653.308827] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 653.319561] Lustre: Skipped 1 previous similar message [ 653.336145] Lustre: 26184:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 653.481400] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 661.795395] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 662.671466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 662.695755] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 662.746835] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 668.597064] Lustre: 25171:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 255 < left 264, rollback = 2 [ 668.616109] Lustre: 25171:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 668.627895] Lustre: 25171:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 668.645265] Lustre: 25171:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 668.654409] Lustre: 25171:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 668.673247] Lustre: 25171:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 684.692378] Lustre: 25171:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 684.714183] Lustre: 25171:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 684.735291] Lustre: 25171:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 684.749680] Lustre: 25171:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 684.758546] Lustre: 25171:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 684.772285] Lustre: 25171:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 690.112902] Lustre: 25171:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 690.122182] Lustre: 25171:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 690.127522] Lustre: 25171:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 690.133760] Lustre: 25171:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 690.139927] Lustre: 25171:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 690.145944] Lustre: 25171:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 690.156098] Lustre: 25171:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 690.162877] Lustre: 25171:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 690.168724] Lustre: 25171:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 690.173878] Lustre: 25171:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 690.180450] Lustre: 25171:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 690.186801] Lustre: 25171:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 693.347748] Lustre: 26405:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 693.360565] Lustre: 26405:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 693.365782] Lustre: 26405:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 693.372493] Lustre: 26405:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 693.379317] Lustre: 26405:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 693.396821] Lustre: 26405:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 693.409161] Lustre: 26405:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 693.423494] Lustre: 26405:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 693.440511] Lustre: 26405:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 693.454790] Lustre: 26405:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 693.480218] Lustre: 26405:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 693.489187] Lustre: 26405:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 698.639360] Lustre: 26405:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 698.660491] Lustre: 26405:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 698.675313] Lustre: 26405:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 698.696809] Lustre: 26405:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 698.702300] Lustre: 26405:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 698.711021] Lustre: 26405:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 698.725979] Lustre: 26405:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 698.741025] Lustre: 26405:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 698.766213] Lustre: 26405:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 698.780210] Lustre: 26405:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 698.794924] Lustre: 26405:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 698.802401] Lustre: 26405:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 707.225129] Lustre: 26405:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 707.237931] Lustre: 26405:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 707.252631] Lustre: 26405:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 707.264860] Lustre: 26405:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 707.273040] Lustre: 26405:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 707.285626] Lustre: 26405:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 707.298359] Lustre: 26405:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 707.314395] Lustre: 26405:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 707.325179] Lustre: 26405:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 707.337968] Lustre: 26405:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 707.345412] Lustre: 26405:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 707.354218] Lustre: 26405:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 723.691481] Lustre: 25171:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 723.708299] Lustre: 25171:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 7 previous similar messages [ 723.721830] Lustre: 25171:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 723.735530] Lustre: 25171:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 723.750252] Lustre: 25171:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 723.767420] Lustre: 25171:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 723.777544] Lustre: 25171:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 723.804737] Lustre: 25171:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 723.821822] Lustre: 25171:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 723.835238] Lustre: 25171:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 723.844329] Lustre: 25171:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 723.853934] Lustre: 25171:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 764.909490] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 764.925052] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 769.457125] Lustre: server umount lustre-OST07c6 complete [ 781.160304] Lustre: server umount lustre-MDT0000 complete [ 800.320381] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [ 803.643906] Key type lgssc unregistered [ 803.975196] LNet: 28743:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 803.993026] LNetError: 28743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 804.018267] LNet: Removed LNI 192.168.204.139@tcp [ 804.901648] Key type .llcrypt unregistered [ 804.904355] Key type ._llcrypt unregistered [ 825.388901] Key type ._llcrypt registered [ 825.391525] Key type .llcrypt registered [ 825.523288] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [ 826.797708] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 826.898417] alg: No test for adler32 (adler32-zlib) [ 827.926498] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 828.175712] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [ 829.871921] Key type lgssc registered [ 831.208724] Lustre: Echo OBD driver; http://www.lustre.org/ [ 840.078817] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 840.095414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 841.728375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 846.519718] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 851.958362] LustreError: 30013:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 851.978096] LustreError: 30013:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 856.920095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 857.063779] LustreError: 30014:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 857.407730] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 862.674807] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 871.276251] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 878.604916] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 886.425454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 886.876358] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 893.921432] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 902.153410] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 911.429170] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 931.824799] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 931.831759] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 931.852232] Lustre: Skipped 1 previous similar message [ 931.853514] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 931.862492] Lustre: Skipped 1 previous similar message [ 935.290367] Lustre: server umount lustre-OST0000 complete [ 942.052375] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 942.070092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 942.082393] Lustre: Skipped 1 previous similar message [ 942.112250] Lustre: Skipped 1 previous similar message [ 947.169383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 947.183345] Lustre: Skipped 1 previous similar message [ 951.277392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 951.287380] Lustre: Skipped 1 previous similar message [ 954.335677] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 954.631226] Lustre: server umount lustre-MDT0000 complete [ 956.387949] LustreError: 30010:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 956.416176] LustreError: 30010:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 958.619384] LustreError: 29994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154480 with bad export cookie 10110299287580501654 [ 958.634437] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 958.641287] LustreError: 29994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 959.172078] Lustre: server umount lustre-MDT0001 complete [ 972.372146] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [ 976.005981] Key type lgssc unregistered [ 976.314427] LNet: 33780:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 976.332036] LNetError: 33780:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 976.358475] LNet: Removed LNI 192.168.204.139@tcp [ 977.437129] Key type .llcrypt unregistered [ 977.442993] Key type ._llcrypt unregistered [ 998.322303] Key type ._llcrypt registered [ 998.326269] Key type .llcrypt registered [ 998.450329] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [ 999.553126] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 999.628516] alg: No test for adler32 (adler32-zlib) [ 1000.756850] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1001.012120] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [ 1002.695992] Key type lgssc registered [ 1004.085615] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1012.112632] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1012.134390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1013.642517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1018.645257] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 1023.970212] LustreError: 35051:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1023.988879] LustreError: 35051:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1027.222508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1027.583593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1032.745709] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 1043.019169] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1050.924129] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1059.582483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1059.901372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1067.022445] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 1076.079420] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1083.449667] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1098.723472] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1098.727109] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1098.728207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1098.737898] Lustre: Skipped 1 previous similar message [ 1103.041527] Lustre: server umount lustre-OST0000 complete [ 1108.975525] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1108.977621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1109.002775] Lustre: Skipped 1 previous similar message [ 1109.020325] Lustre: Skipped 2 previous similar messages [ 1113.293408] Lustre: server umount lustre-MDT0000 complete [ 1114.080567] LustreError: 35052:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1114.117386] LustreError: 35052:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1117.571527] LustreError: 36898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154639 with bad export cookie 11513479477680287951 [ 1117.582340] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1117.585155] LustreError: 36898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1117.969617] Lustre: server umount lustre-MDT0001 complete [ 1129.411957] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [ 1132.344624] Key type lgssc unregistered [ 1132.596649] LNet: 38609:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1132.612023] LNetError: 38609:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1132.633683] LNet: Removed LNI 192.168.204.139@tcp [ 1133.562751] Key type .llcrypt unregistered [ 1133.569931] Key type ._llcrypt unregistered [ 1157.936819] Key type ._llcrypt registered [ 1157.942583] Key type .llcrypt registered [ 1158.044770] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [ 1159.395997] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1159.425883] alg: No test for adler32 (adler32-zlib) [ 1160.637519] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1160.974096] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [ 1162.663241] Key type lgssc registered [ 1163.910571] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1172.669663] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1172.699496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1174.188358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1178.803320] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 1184.740180] LustreError: 40025:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1184.755835] LustreError: 40025:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1187.904115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1188.286608] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1193.307377] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 1200.662336] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1207.001707] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1214.065421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1214.484635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1219.642979] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1221.136530] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 1228.651726] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1239.017386] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1239.018877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1239.030571] Lustre: Skipped 1 previous similar message [ 1239.043373] Lustre: Skipped 2 previous similar messages [ 1240.401746] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.39@tcp (stopping) [ 1243.405517] LustreError: 42604:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x736d61726170:0x3:0x0].0x0 (00000000faa39a91) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 1243.500105] Lustre: server umount lustre-MDT0000 complete [ 1244.138615] LustreError: 40022:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1244.158235] LustreError: 40022:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1245.490624] LustreError: 40021:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1246.991713] LustreError: 40005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154768 with bad export cookie 3145028673123366579 [ 1247.002138] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1247.006280] LustreError: 40005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1249.248589] LustreError: 40022:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1249.250201] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1249.297214] Lustre: Skipped 1 previous similar message [ 1249.301138] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1252.256503] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1252.264281] Lustre: Skipped 1 previous similar message [ 1253.512251] Lustre: server umount lustre-MDT0001 complete [ 1275.225451] Lustre: server umount lustre-OST0000 complete [ 1290.923378] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [ 1293.718888] Key type lgssc unregistered [ 1293.985700] LNet: 43889:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1293.995155] LNetError: 43889:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1294.029881] LNet: Removed LNI 192.168.204.139@tcp [ 1294.840488] Key type .llcrypt unregistered [ 1294.845226] Key type ._llcrypt unregistered [ 1319.705618] Key type ._llcrypt registered [ 1319.707858] Key type .llcrypt registered [ 1319.881345] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [ 1320.986465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1321.100996] alg: No test for adler32 (adler32-zlib) [ 1322.323464] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1322.630287] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [ 1324.335752] Key type lgssc registered [ 1325.721549] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1333.398790] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1333.422895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1334.950560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1339.751377] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 1344.486543] LustreError: 45310:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1344.506441] LustreError: 45310:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1348.490156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1348.994241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1353.896677] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 1361.889867] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1367.101133] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1374.129399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1374.464798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1379.631755] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1380.929388] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 1388.289644] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1399.270117] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1399.271311] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1399.287072] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1400.148718] Lustre: lustre-OST0000: Not available for connect from 192.168.204.39@tcp (stopping) [ 1400.153344] Lustre: Skipped 1 previous similar message [ 1403.640396] Lustre: server umount lustre-OST0000 complete [ 1424.879648] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1424.886306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1424.899807] Lustre: Skipped 2 previous similar messages [ 1424.913382] Lustre: Skipped 1 previous similar message [ 1429.429215] Lustre: server umount lustre-MDT0000 complete [ 1429.996421] LustreError: 45310:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1430.031286] LustreError: 45310:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1433.519846] LustreError: 45290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154955 with bad export cookie 14300801150161649291 [ 1433.535211] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1433.550570] LustreError: 45290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1440.132749] Lustre: server umount lustre-MDT0001 complete [ 1452.793826] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [ 1455.760786] Key type lgssc unregistered [ 1456.069535] LNet: 49020:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1456.080528] LNetError: 49020:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1456.122577] LNet: Removed LNI 192.168.204.139@tcp [ 1457.213152] Key type .llcrypt unregistered [ 1457.216906] Key type ._llcrypt unregistered [ 1485.136825] Key type ._llcrypt registered [ 1485.140987] Key type .llcrypt registered [ 1485.261070] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [ 1486.321930] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1486.529715] alg: No test for adler32 (adler32-zlib) [ 1487.561989] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1487.739249] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [ 1489.375259] Key type lgssc registered [ 1490.571899] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1498.416279] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1498.445834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1500.031959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1505.296903] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 1509.350335] LustreError: 50438:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1509.370940] LustreError: 50438:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1514.379352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1514.464020] LustreError: 50437:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1514.848838] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1519.870269] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 1527.034043] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1532.855340] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1539.500694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1539.950432] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1545.092631] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1546.288216] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 1553.712315] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1562.336898] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1567.084877] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1579.136206] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-31 1h44m59s [ 1580.473016] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1583.011732] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1583.015410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1583.039660] Lustre: Skipped 1 previous similar message [ 1583.058934] Lustre: Skipped 2 previous similar messages [ 1584.099490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1584.104750] Lustre: Skipped 2 previous similar messages [ 1586.039455] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.39@tcp (stopping) [ 1589.010380] Lustre: server umount lustre-MDT0000 complete [ 1590.242341] LustreError: 50437:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1590.272558] LustreError: 50437:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1592.893673] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1593.645356] LustreError: 53125:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1599.801714] LustreError: 53075:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1599.819454] LustreError: 53075:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1600.212811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1600.462509] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1600.863033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1600.957596] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1601.924309] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1605.754112] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 1609.506236] Lustre: DEBUG MARKER: Wait for df (36712) ... [ 1611.187052] Lustre: DEBUG MARKER: done [ 1615.843978] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1615.853352] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1615.866230] Lustre: Skipped 1 previous similar message [ 1615.875678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1615.882849] Lustre: Skipped 1 previous similar message [ 1616.359802] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1622.505972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1622.517528] Lustre: Skipped 3 previous similar messages [ 1629.151109] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1629.336276] Lustre: server umount lustre-OST0000 complete [ 1637.878463] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1637.891277] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1637.898087] Lustre: Skipped 1 previous similar message [ 1637.900038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1637.900045] Lustre: Skipped 3 previous similar messages [ 1640.210501] Lustre: server umount lustre-MDT0000 complete [ 1643.894043] LustreError: 50418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155165 with bad export cookie 10407531728600840928 [ 1643.896692] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1643.906842] LustreError: 50418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1644.345699] Lustre: server umount lustre-MDT0001 complete [ 1654.323951] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [ 1656.933772] Key type lgssc unregistered [ 1657.189472] LNet: 55371:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1657.198118] LNetError: 55371:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1657.214842] LNet: Removed LNI 192.168.204.139@tcp [ 1658.020393] Key type .llcrypt unregistered [ 1658.024692] Key type ._llcrypt unregistered [ 1697.186959] Key type ._llcrypt registered [ 1697.191142] Key type .llcrypt registered [ 1697.301562] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [ 1698.984451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1698.999478] alg: No test for adler32 (adler32-zlib) [ 1700.062622] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1700.287280] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [ 1701.935219] Key type lgssc registered [ 1703.098764] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1710.380266] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1710.405825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1711.714893] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1711.740257] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1711.913628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1716.266909] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 1724.550127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1724.665863] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1724.713164] Lustre: 57574:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1724.960835] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1728.860438] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 1736.366940] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1741.825031] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1747.677217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1747.928809] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1748.088728] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1753.679369] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 1756.866415] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1760.525220] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1767.180740] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-31 1h48m07s [ 1768.886375] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1775.636256] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1778.222801] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1778.284339] LustreError: 59442:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1798.534128] Lustre: lustre-MDT0000: Client 52a8f066-cc8c-44b0-9e61-c46f46d10537 (at 192.168.204.39@tcp) reconnecting [ 1818.319127] LustreError: 59442:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1820.591598] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1823.710930] Lustre: DEBUG MARKER: done [ 1833.449677] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1833.464353] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1833.484056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1839.076554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1839.087943] Lustre: Skipped 2 previous similar messages [ 1844.191135] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1844.228090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1844.238468] Lustre: Skipped 1 previous similar message [ 1844.580850] Lustre: server umount lustre-OST0000 complete [ 1849.316701] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1849.321134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1849.362781] Lustre: Skipped 2 previous similar messages [ 1849.372714] Lustre: Skipped 1 previous similar message [ 1854.430050] Lustre: server umount lustre-MDT0000 complete [ 1857.675741] LustreError: 57565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155379 with bad export cookie 14236839069331882548 [ 1857.680809] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1857.688415] LustreError: 57565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1858.026948] Lustre: server umount lustre-MDT0001 complete [ 1871.484400] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [ 1874.724874] Key type lgssc unregistered [ 1875.122681] LNet: 61902:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1875.131934] LNetError: 61902:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1875.151834] LNet: Removed LNI 192.168.204.139@tcp [ 1876.239904] Key type .llcrypt unregistered [ 1876.245304] Key type ._llcrypt unregistered [ 1920.095446] Key type ._llcrypt registered [ 1920.099156] Key type .llcrypt registered [ 1920.289744] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [ 1921.962562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1922.013808] alg: No test for adler32 (adler32-zlib) [ 1923.258596] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1923.490541] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [ 1925.191226] Key type lgssc registered [ 1926.363646] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1929.306351] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1929.333947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1930.567482] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1930.585689] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1930.729274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1931.605603] Lustre: Failing over lustre-MDT0000 [ 1931.910127] Lustre: server umount lustre-MDT0000 complete [ 1938.495346] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 01:50:58 (1788155458) [ 1944.937439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1945.472390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1950.144077] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 1957.955617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1958.116872] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1958.152253] Lustre: 64704:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1958.412422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1962.844675] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 1970.301839] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1976.075351] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1982.837648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1983.041865] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1983.273457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1989.305965] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 1991.301248] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1996.261913] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2002.282532] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2009.571590] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2009.582777] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2009.609633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2012.142479] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2012.143807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2012.158857] Lustre: Skipped 1 previous similar message [ 2012.168433] Lustre: Skipped 1 previous similar message [ 2017.249419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2017.256378] Lustre: Skipped 2 previous similar messages [ 2022.369138] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2022.375599] Lustre: Skipped 2 previous similar messages [ 2023.391627] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2023.614615] Lustre: server umount lustre-MDT0000 complete [ 2027.625037] LustreError: 66557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155549 with bad export cookie 7101724469163707636 [ 2027.635297] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2027.641921] LustreError: 66557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2028.003599] Lustre: server umount lustre-MDT0001 complete [ 2031.215342] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2035.817786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2045.506629] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2048.991618] Lustre: 63626:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788155554/real 1788155554] req@000000008e5eb9cc x1875016883939456/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788155570 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2049.021520] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2052.420236] LustreError: 66551:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2052.450716] LustreError: 66551:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2052.493476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2052.584488] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2053.090511] Lustre: 63624:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788155559/real 1788155559] req@0000000041f3faee x1875016883939712/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788155575 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2054.507543] LustreError: 66552:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2057.575772] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2057.578973] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 2057.581859] LustreError: 66551:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2058.593350] Lustre: 63624:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788155564/real 1788155564] req@0000000048cfd0c2 x1875016883939840/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788155580 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2063.712349] Lustre: 63627:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788155569/real 1788155569] req@000000009751be16 x1875016883939968/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788155585 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2063.841053] LustreError: 68472:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2063.855266] LustreError: 68472:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2065.124372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2065.434327] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2069.382785] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 2070.528133] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2076.599754] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2081.626552] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2089.738379] Lustre: server umount lustre-MDT0000 complete [ 2090.977046] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2090.984847] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2090.994178] LustreError: 68473:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2091.000801] Lustre: Skipped 2 previous similar messages [ 2091.030505] LustreError: 68473:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2093.379048] LustreError: 64696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155615 with bad export cookie 7101724469163709911 [ 2093.380525] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2093.387084] LustreError: 64696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2093.649952] Lustre: server umount lustre-MDT0001 complete [ 2099.251330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2108.710778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2112.479293] Lustre: 63627:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788155618/real 1788155618] req@00000000f44cbc21 x1875016883965440/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788155634 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2112.507746] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2118.687873] LustreError: 63623:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000c8b81027 x1875016883966592/t0(0) o250->MGC192.168.204.139@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2119.004631] LustreError: 69206:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2119.079989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2119.138201] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2122.846531] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 2124.623160] Lustre: 63626:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788155630/real 1788155630] req@000000001d76eb1e x1875016883966080/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788155646 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2124.662086] Lustre: 63626:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 2131.261608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2131.462744] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2136.105100] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 2137.068608] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2144.814662] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2151.816889] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2162.658941] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2162.681858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2164.825040] Lustre: server umount lustre-MDT0000 complete [ 2167.780600] LustreError: 66551:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2167.812864] LustreError: 66551:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 2168.781674] LustreError: 64696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155690 with bad export cookie 7101724469163711290 [ 2168.790235] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2168.801472] LustreError: 64696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2169.229136] Lustre: server umount lustre-MDT0001 complete [ 2175.810780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2179.301639] Lustre: DEBUG MARKER: files compared the same [ 2190.492077] Lustre: server umount lustre-OST0000 complete [ 2211.436167] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [ 2214.435946] Key type lgssc unregistered [ 2214.741076] LNet: 74431:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2214.747667] LNetError: 74431:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2214.773164] LNet: Removed LNI 192.168.204.139@tcp [ 2215.754981] Key type .llcrypt unregistered [ 2215.759437] Key type ._llcrypt unregistered [ 2243.302798] Key type ._llcrypt registered [ 2243.306397] Key type .llcrypt registered [ 2243.445890] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [ 2244.856430] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2244.986021] alg: No test for adler32 (adler32-zlib) [ 2246.326489] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2246.693621] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [ 2248.415282] Key type lgssc registered [ 2249.520489] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2257.869461] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2257.896450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2259.581258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2265.366848] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug +malloc all [ 2269.156455] LustreError: 75850:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2269.173469] LustreError: 75850:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2274.157819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2274.275953] LustreError: 75851:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2274.544629] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2278.337775] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug +malloc all [ 2285.112302] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2291.087662] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2298.957487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2299.272836] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2304.435039] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2305.338612] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug +malloc all [ 2312.485859] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2324.966317] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2324.974114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2324.974581] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2324.979466] Lustre: Skipped 1 previous similar message [ 2324.999511] Lustre: Skipped 1 previous similar message [ 2327.617933] Lustre: server umount lustre-OST0000 complete [ 2331.107705] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2331.121387] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2331.136752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2335.209978] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2335.211639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2335.245399] Lustre: Skipped 1 previous similar message [ 2337.293690] Lustre: server umount lustre-MDT0000 complete [ 2340.322108] LustreError: 78347:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2340.344792] LustreError: 78347:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2341.220502] LustreError: 75832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155863 with bad export cookie 16809485382472637655 [ 2341.223724] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2341.234375] LustreError: 75832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2341.642405] Lustre: server umount lustre-MDT0001 complete [ 2351.945434] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [ 2354.810630] Key type lgssc unregistered [ 2355.165309] LNet: 79381:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2355.184836] LNetError: 79381:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2355.204778] LNet: Removed LNI 192.168.204.139@tcp [ 2356.311247] Key type .llcrypt unregistered [ 2356.318502] Key type ._llcrypt unregistered [ 2379.610898] Key type ._llcrypt registered [ 2379.612378] Key type .llcrypt registered [ 2379.728163] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [ 2380.730310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2380.744824] alg: No test for adler32 (adler32-zlib) [ 2381.952738] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2382.182291] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [ 2383.887809] Key type lgssc registered [ 2384.921522] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2392.824678] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2392.852890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2408.415586] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2413.535474] LustreError: 80774:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.139@tcp: failed processing log, type 4: rc = -110 [ 2440.223658] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2447.164479] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 2455.675809] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2463.342420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2463.673542] Lustre: *** cfs_fail_loc=706, val=0*** [ 2464.112710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2464.174375] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2469.335665] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 2469.344941] LustreError: 80799:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2469.372032] LustreError: 80799:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2474.465082] LustreError: 80800:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2474.491518] LustreError: 80800:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2478.596042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2479.014879] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2483.607898] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 2491.055823] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2497.468494] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2500.066263] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2500.080169] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2500.100118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2504.679533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2504.681337] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2504.686262] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2506.005402] Lustre: server umount lustre-OST0000 complete [ 2509.793228] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2509.796676] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2509.800705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2509.800714] Lustre: Skipped 1 previous similar message [ 2509.838482] Lustre: Skipped 1 previous similar message [ 2514.915167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2514.925041] Lustre: Skipped 2 previous similar messages [ 2516.068148] Lustre: server umount lustre-MDT0000 complete [ 2519.899855] LustreError: 82599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156041 with bad export cookie 15557368279161895647 [ 2519.905627] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2519.932296] LustreError: 82599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2520.397844] Lustre: server umount lustre-MDT0001 complete [ 2532.054481] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [ 2534.801769] Key type lgssc unregistered [ 2535.129083] LNet: 84388:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2535.140595] LNetError: 84388:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2535.165181] LNet: Removed LNI 192.168.204.139@tcp [ 2536.155370] Key type .llcrypt unregistered [ 2536.161088] Key type ._llcrypt unregistered [ 2560.459624] Key type ._llcrypt registered [ 2560.464325] Key type .llcrypt registered [ 2560.627392] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [ 2562.010867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2562.090575] alg: No test for adler32 (adler32-zlib) [ 2563.293781] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2563.589425] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [ 2565.247725] Key type lgssc registered [ 2566.336325] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2574.240564] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2574.267463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2580.862554] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 2588.499830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2589.080734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2593.756345] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 2594.277512] LustreError: 86497:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2594.308654] LustreError: 86497:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2598.383764] LustreError: 86501:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2598.414326] LustreError: 86501:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2603.504266] LustreError: 86500:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2603.510636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2604.088568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2609.135256] LustreError: 86496:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2609.181779] LustreError: 86496:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2612.147669] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 2613.234453] LustreError: 86501:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2613.262367] LustreError: 86501:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2621.861730] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2621.879647] Lustre: MGS: Client 98522673-e0e2-48de-94d2-5f37304779ac (at 0@lo) reconnecting [ 2621.916533] LustreError: 86511:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2622.220546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2622.293633] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2627.193105] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 2642.922736] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2642.923615] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2642.926508] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2642.942910] Lustre: Skipped 1 previous similar message [ 2648.070590] LustreError: 87225:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2648.090585] LustreError: 87225:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2648.168296] Lustre: server umount lustre-OST0000 complete [ 2653.154883] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2653.160494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2653.169866] Lustre: Skipped 1 previous similar message [ 2653.185423] Lustre: Skipped 2 previous similar messages [ 2658.284792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2658.302784] Lustre: Skipped 1 previous similar message [ 2658.485897] Lustre: server umount lustre-MDT0000 complete [ 2662.350358] Lustre: server umount lustre-MDT0001 complete [ 2666.335524] Lustre: server umount MGS complete [ 2677.988325] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 02:03:18 (1788156198) [ 2708.151754] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_hostid [ 2718.457875] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [ 2753.226061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2758.569268] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 2769.539093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2769.892283] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2769.954769] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2770.124930] Lustre: lustre-MDT0001: new disk, initializing [ 2770.216254] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2770.258886] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2770.272863] Lustre: 93690:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2770.288718] LustreError: 93690:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2775.551539] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 2787.461603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2787.760978] Lustre: lustre-OST0000: new disk, initializing [ 2787.764402] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2787.771531] Lustre: 94456:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2787.867971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2794.635614] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 2795.620393] Lustre: 94795:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000000d6a482e x1875017555026816/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788156333 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2804.839413] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2811.485251] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2811.510828] Lustre: MGS: Client e5902fb5-752a-48fe-bb93-7ed2237530b7 (at 0@lo) reconnecting [ 2811.596806] Lustre: 95501:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2811.615492] LustreError: 95501:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 2811.627950] LustreError: 95501:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2811.655195] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2811.757383] Lustre: lustre-MDT0000: new disk, initializing [ 2811.902587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2811.955114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2811.983069] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2813.024722] Lustre: 95525:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000005710ef7e x1875017555033728/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788156350 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2816.351468] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 2818.099657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2818.116707] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2818.256370] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2843.107487] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2843.118344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2843.126273] Lustre: Skipped 1 previous similar message [ 2843.139135] Lustre: Skipped 1 previous similar message [ 2846.681398] Lustre: server umount lustre-OST0000 complete [ 2853.344329] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2853.347836] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2853.353834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2853.373728] Lustre: Skipped 1 previous similar message [ 2863.585077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2863.599788] Lustre: Skipped 4 previous similar messages [ 2865.121491] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2865.237854] Lustre: server umount lustre-MDT0000 complete [ 2868.704539] LustreError: 95951:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2876.019835] Lustre: server umount lustre-MDT0001 complete [ 2879.813921] Lustre: server umount MGS complete [ 2891.657144] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 02:06:51 (1788156411) [ 2907.723562] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [ 2910.547888] Key type lgssc unregistered [ 2910.842912] LNet: 98023:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2910.853507] LNetError: 98023:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2910.874055] LNet: Removed LNI 192.168.204.139@tcp [ 2911.677983] Key type .llcrypt unregistered [ 2911.680943] Key type ._llcrypt unregistered [ 2929.094323] Key type ._llcrypt registered [ 2929.108410] Key type .llcrypt registered [ 2929.237249] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [ 2930.623624] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2930.639450] alg: No test for adler32 (adler32-zlib) [ 2931.719428] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2932.018898] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [ 2933.679175] Key type lgssc registered [ 2934.876731] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2941.850213] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2941.887228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2943.386502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2947.068199] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 2953.699535] LustreError: 99246:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2953.706643] LustreError: 99246:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2954.323616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2954.601367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2958.061806] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 2964.457116] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2969.829181] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2976.338689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2976.694241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2983.103560] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 2989.009529] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2990.882991] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2999.269777] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2999.271185] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2999.281978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2999.288482] Lustre: Skipped 1 previous similar message [ 3004.329784] Lustre: server umount lustre-OST0000 complete [ 3009.504512] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3009.538746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3009.543750] Lustre: Skipped 2 previous similar messages [ 3014.630178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3014.638797] Lustre: Skipped 1 previous similar message [ 3015.103243] Lustre: server umount lustre-MDT0000 complete [ 3019.750107] LustreError: 99245:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3019.782546] LustreError: 99245:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3019.931321] LustreError: 99226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156541 with bad export cookie 17793394911527341017 [ 3019.937210] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3019.954188] LustreError: 99226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3020.399956] Lustre: server umount lustre-MDT0001 complete [ 3034.218772] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [ 3038.088057] Key type lgssc unregistered [ 3038.381860] LNet: 102779:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3038.390039] LNetError: 102779:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3038.402156] LNet: Removed LNI 192.168.204.139@tcp [ 3039.314896] Key type .llcrypt unregistered [ 3039.321770] Key type ._llcrypt unregistered [ 3056.863602] Key type ._llcrypt registered [ 3056.865275] Key type .llcrypt registered [ 3056.985484] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [ 3058.060434] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3058.123630] alg: No test for adler32 (adler32-zlib) [ 3059.115441] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 3059.333631] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [ 3061.016058] Key type lgssc registered [ 3062.162606] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3068.543483] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3068.555739] LustreError: 103856:0:(tgt_mount.c:2644:server_fill_super()) cfs_race id 716 sleeping [ 3068.607378] LustreError: 103862:0:(tgt_mount.c:2644:server_fill_super()) cfs_fail_race id 716 waking [ 3068.610789] LustreError: 103856:0:(tgt_mount.c:2644:server_fill_super()) cfs_fail_race id 716 awake: rc=4961 [ 3068.623101] LustreError: 103856:0:(tgt_mount.c:2656:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3068.640763] LustreError: 103856:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3068.662551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3070.325636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3077.863735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3078.124141] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3082.863587] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 3088.776380] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing lsmod [ 3091.553406] LustreError: 105075:0:(tgt_mount.c:2644:server_fill_super()) cfs_race id 716 sleeping [ 3091.584495] LustreError: 105076:0:(tgt_mount.c:2644:server_fill_super()) cfs_fail_race id 716 waking [ 3091.591838] LustreError: 105075:0:(tgt_mount.c:2644:server_fill_super()) cfs_fail_race id 716 awake: rc=4971 [ 3091.600958] LustreError: 105075:0:(tgt_mount.c:2656:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3091.610364] LustreError: 105075:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3091.636703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3092.030685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3097.058037] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3097.058776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3097.068789] Lustre: Skipped 2 previous similar messages [ 3097.075315] Lustre: Skipped 1 previous similar message [ 3102.176756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3102.184199] Lustre: Skipped 2 previous similar messages [ 3103.206995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3103.214115] Lustre: Skipped 2 previous similar messages [ 3108.329658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3108.343719] Lustre: Skipped 2 previous similar messages [ 3109.345408] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3109.555528] Lustre: server umount lustre-MDT0000 complete [ 3113.444179] LustreError: 103899:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3113.461818] LustreError: 103899:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3113.960991] LustreError: 103875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156635 with bad export cookie 376520070207829215 [ 3113.964408] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3113.978098] LustreError: 103875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3115.551271] LustreError: 104330:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000073fdb27d x1875018075101952/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3115.843962] Lustre: server umount lustre-MDT0001 complete [ 3130.068212] Lustre: server umount lustre-OST0000 complete [ 3139.733042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3140.111185] LustreError: 106143:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3140.194224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3145.189261] LustreError: 106144:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3145.288266] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 3150.305374] LustreError: 106143:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3153.819210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3154.296037] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3158.740327] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 3167.337705] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3173.956152] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3181.685949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3182.103144] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3185.216125] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3189.748313] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 3199.071653] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3209.696721] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3209.706133] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3209.715753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3209.756185] Lustre: Skipped 1 previous similar message [ 3213.940503] Lustre: server umount lustre-OST0000 complete [ 3219.938056] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3219.947251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3219.970260] Lustre: Skipped 1 previous similar message [ 3219.973826] Lustre: Skipped 1 previous similar message [ 3225.147934] Lustre: server umount lustre-MDT0000 complete [ 3229.218565] LustreError: 106143:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3229.256737] LustreError: 106143:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3229.939862] LustreError: 106125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156751 with bad export cookie 376520070207829908 [ 3229.956465] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3229.956763] LustreError: 106125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3230.787395] Lustre: server umount lustre-MDT0001 complete [ 3242.428678] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [ 3244.991497] Key type lgssc unregistered [ 3245.354600] LNet: 109678:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3245.359509] LNetError: 109678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3245.391490] LNet: Removed LNI 192.168.204.139@tcp [ 3246.285961] Key type .llcrypt unregistered [ 3246.289302] Key type ._llcrypt unregistered [ 3269.199616] Key type ._llcrypt registered [ 3269.205047] Key type .llcrypt registered [ 3269.347543] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [ 3270.596084] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3270.628591] alg: No test for adler32 (adler32-zlib) [ 3271.777810] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 3272.003704] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [ 3273.656368] Key type lgssc registered [ 3274.943436] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3284.452922] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3284.467374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3285.924530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3291.328375] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 3295.207165] LustreError: 111095:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3295.243025] LustreError: 111095:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3300.333602] LustreError: 111096:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3301.268889] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3301.615574] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3306.589932] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 3315.912808] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3323.656562] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3330.690346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3331.142830] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3337.782284] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 3346.106443] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3355.778974] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3360.824815] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3364.127991] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3368.419360] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3368.430362] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3368.447455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3370.049713] Lustre: server umount lustre-OST0000 complete [ 3376.619976] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3376.625070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3376.644788] Lustre: Skipped 1 previous similar message [ 3376.672659] Lustre: Skipped 1 previous similar message [ 3381.736078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3381.749040] Lustre: Skipped 1 previous similar message [ 3384.883840] Lustre: server umount lustre-MDT0000 complete [ 3386.850813] LustreError: 111091:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3386.867838] LustreError: 111091:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3388.563234] LustreError: 111830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156910 with bad export cookie 18032679210187830502 [ 3388.572758] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3388.578636] LustreError: 111830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3389.062862] Lustre: server umount lustre-MDT0001 complete [ 3401.842497] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [ 3405.462289] Key type lgssc unregistered [ 3405.743675] LNet: 114757:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3405.751600] LNetError: 114757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3405.787776] LNet: Removed LNI 192.168.204.139@tcp [ 3406.902831] Key type .llcrypt unregistered [ 3406.905050] Key type ._llcrypt unregistered [ 3428.334902] Key type ._llcrypt registered [ 3428.337373] Key type .llcrypt registered [ 3428.507978] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [ 3430.267748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3430.332730] alg: No test for adler32 (adler32-zlib) [ 3431.605051] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 3432.044286] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [ 3433.738208] Key type lgssc registered [ 3435.345901] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3445.483708] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3445.549852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3447.290947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3452.382383] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 3457.505753] LustreError: 115981:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3457.561086] LustreError: 115981:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3461.976493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3462.486367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3467.941971] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 3477.853439] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3485.127728] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3492.011444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3492.343343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3497.471199] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3499.043974] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 3506.365578] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3517.926681] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3517.930290] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3517.950540] Lustre: Skipped 1 previous similar message [ 3517.957827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3517.968142] Lustre: Skipped 1 previous similar message [ 3521.138108] Lustre: server umount lustre-OST0000 complete [ 3528.160539] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3528.184082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3528.192240] Lustre: Skipped 1 previous similar message [ 3531.215573] Lustre: server umount lustre-MDT0000 complete [ 3533.282094] LustreError: 115982:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3533.312797] LustreError: 115982:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3535.610812] LustreError: 117825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157057 with bad export cookie 4592621057014391301 [ 3535.626046] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3535.638221] LustreError: 117825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3536.184496] Lustre: server umount lustre-MDT0001 complete [ 3547.987759] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [ 3551.021859] Key type lgssc unregistered [ 3551.288718] LNet: 119514:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3551.301553] LNetError: 119514:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3551.314977] LNet: Removed LNI 192.168.204.139@tcp [ 3552.171442] Key type .llcrypt unregistered [ 3552.174151] Key type ._llcrypt unregistered [ 3578.504331] Key type ._llcrypt registered [ 3578.511574] Key type .llcrypt registered [ 3578.679522] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [ 3579.954758] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3580.006926] alg: No test for adler32 (adler32-zlib) [ 3581.429747] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 3581.768911] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [ 3583.551992] Key type lgssc registered [ 3585.458804] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3594.497955] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3594.524727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3595.867377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3600.765920] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 3605.986303] LustreError: 120932:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3606.009209] LustreError: 120932:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3609.487509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3609.878810] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3614.864854] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 3621.274664] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3626.851124] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3634.539938] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3634.950642] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3641.107302] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3643.881978] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 3655.548898] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3664.307526] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3668.117370] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3668.131159] Lustre: Skipped 2 previous similar messages [ 3669.559557] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3669.609380] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3672.529932] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3672.534984] Lustre: Skipped 2 previous similar messages [ 3676.430928] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3676.447273] Lustre: Skipped 1 previous similar message [ 3684.252092] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.39@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3684.271946] Lustre: Skipped 2 previous similar messages [ 3684.615782] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.39@tcp 192.168.0.[2,10]@tcp [ 3690.031546] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.39/24@tcp in log lustre-MDT0000 [ 3690.042608] Lustre: Skipped 2 previous similar messages [ 3691.598077] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.39/24@tcp [ 3691.612367] Lustre: Skipped 1 previous similar message [ 3697.633996] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3697.641376] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3697.653057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3700.704352] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3700.719597] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3700.722506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3703.995330] Lustre: server umount lustre-OST0000 complete [ 3710.952727] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3710.976457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3710.984189] Lustre: Skipped 1 previous similar message [ 3714.950636] Lustre: server umount lustre-MDT0000 complete [ 3716.064636] LustreError: 120933:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3716.106697] LustreError: 120933:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3719.116882] LustreError: 120913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157241 with bad export cookie 8036244080957919301 [ 3719.119234] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3719.136542] LustreError: 120913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3719.611427] Lustre: server umount lustre-MDT0001 complete [ 3729.836322] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [ 3733.048526] Key type lgssc unregistered [ 3733.492086] LNet: 125640:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3733.507286] LNetError: 125640:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3733.536707] LNet: Removed LNI 192.168.204.139@tcp [ 3734.893588] Key type .llcrypt unregistered [ 3734.898143] Key type ._llcrypt unregistered [ 3770.998323] Key type ._llcrypt registered [ 3771.000057] Key type .llcrypt registered [ 3771.184655] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [ 3772.561233] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3772.597174] alg: No test for adler32 (adler32-zlib) [ 3773.661577] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 3774.120759] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [ 3775.847282] Key type lgssc registered [ 3777.736941] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3786.947715] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3786.986260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3788.582791] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3788.616083] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.39/24@tcp [ 3788.637259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3793.423676] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 3799.012192] LustreError: 127205:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3799.030538] LustreError: 127205:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3803.253231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3803.598216] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3803.638588] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.39/24@tcp [ 3803.713919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3808.356276] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 3816.134322] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3822.470024] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3828.953222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3829.315279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3833.438996] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3836.165617] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 3843.705044] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3857.900973] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3857.910997] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3857.925668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3859.428344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3859.432037] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3859.435049] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3863.435574] Lustre: server umount lustre-OST0000 complete [ 3869.664739] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3869.685150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3869.693538] Lustre: Skipped 2 previous similar messages [ 3872.921710] Lustre: server umount lustre-MDT0000 complete [ 3874.785568] LustreError: 127199:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3874.799231] LustreError: 127199:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3875.928901] LustreError: 127184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157397 with bad export cookie 13697680777471882356 [ 3875.929780] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3875.938274] LustreError: 127184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3876.226282] Lustre: server umount lustre-MDT0001 complete [ 3886.024942] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [ 3888.565767] Key type lgssc unregistered [ 3888.800662] LNet: 130794:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3888.805807] LNetError: 130794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3888.814889] LNet: Removed LNI 192.168.204.139@tcp [ 3889.774163] Key type .llcrypt unregistered [ 3889.781356] Key type ._llcrypt unregistered [ 3918.114888] Key type ._llcrypt registered [ 3918.118833] Key type .llcrypt registered [ 3918.271933] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_hostid [ 3933.169530] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [ 3934.431101] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3934.603166] alg: No test for adler32 (adler32-zlib) [ 3935.832804] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 3936.133652] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [ 3937.863184] Key type lgssc registered [ 3938.860851] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3976.579776] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3976.617347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3977.917518] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3977.954410] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3978.040690] Lustre: lustre-MDT0000: new disk, initializing [ 3978.121759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3978.146899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3982.954783] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 3994.073251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3994.176083] Lustre: 134196:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3994.253592] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3994.260230] Lustre: Skipped 1 previous similar message [ 3994.388914] Lustre: lustre-MDT0001: new disk, initializing [ 3994.554211] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3994.625788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3994.646388] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3999.368480] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 4008.065758] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4013.616140] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4019.390751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4019.645750] Lustre: lustre-OST0000: new disk, initializing [ 4019.650250] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4019.658299] Lustre: 136163:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4019.766787] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4025.435867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4025.450440] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4025.583268] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4027.002920] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 4036.524366] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4041.570072] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4041.788325] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4047.563114] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4047.931406] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4051.426307] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4051.446912] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4051.465205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4051.477604] Lustre: Skipped 1 previous similar message [ 4056.348389] Lustre: server umount lustre-OST0000 complete [ 4061.155206] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4061.160396] LustreError: Skipped 1 previous similar message [ 4061.162272] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4061.179034] Lustre: Skipped 1 previous similar message [ 4061.190147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4066.526350] Lustre: server umount lustre-MDT0000 complete [ 4070.918123] LustreError: 134998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157592 with bad export cookie 4664276960857062838 [ 4070.926927] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4070.931319] LustreError: 134998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4071.458036] Lustre: server umount lustre-MDT0001 complete [ 4080.749081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4081.403610] LustreError: 138380:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4081.424188] LustreError: 138380:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4081.492699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4086.272429] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 4086.768476] LustreError: 138381:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4091.872372] LustreError: 138380:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4095.226648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4095.698793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4100.514398] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 4109.151160] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4114.925860] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4122.836787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4123.272364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4129.532550] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 4137.829471] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4142.529229] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4142.722721] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4148.020966] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4148.350199] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4157.303943] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4157.508986] Lustre: lustre-OST0001: new disk, initializing [ 4157.521463] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4157.527771] Lustre: 141751:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4157.607736] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4163.662134] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 4164.727203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4164.737477] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4164.834315] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4172.308961] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4172.476124] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4176.717946] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4177.009737] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4182.535428] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9fcac69e6800.ost_server_uuid 50 [ 4184.737904] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9fcac69e6800.ost_server_uuid in FULL state after 0 sec [ 4195.818438] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4195.819606] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4195.840345] LustreError: Skipped 1 previous similar message [ 4195.842277] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4195.842286] Lustre: Skipped 1 previous similar message [ 4195.881763] Lustre: Skipped 2 previous similar messages [ 4196.245446] Lustre: server umount lustre-OST0001 complete [ 4200.928274] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4200.936524] LustreError: 140204:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4200.944589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4200.946090] Lustre: Skipped 1 previous similar message [ 4200.948025] Lustre: Skipped 1 previous similar message [ 4200.975682] LustreError: 140204:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4206.053771] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4206.055222] LustreError: 140204:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4206.062651] Lustre: Skipped 2 previous similar messages [ 4206.077310] LustreError: 140204:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4206.423572] Lustre: server umount lustre-OST0000 complete [ 4211.169871] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4211.200909] Lustre: Skipped 1 previous similar message [ 4216.291058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4216.295113] Lustre: Skipped 3 previous similar messages [ 4216.980071] Lustre: server umount lustre-MDT0000 complete [ 4221.371793] LustreError: 138362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157743 with bad export cookie 4664276960857063720 [ 4221.372737] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4221.396102] LustreError: 138362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4221.921775] Lustre: server umount lustre-MDT0001 complete [ 4245.285770] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [ 4248.351157] Key type lgssc unregistered [ 4248.613488] LNet: 144756:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4248.619320] LNetError: 144756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4248.648724] LNet: Removed LNI 192.168.204.139@tcp [ 4249.823577] Key type .llcrypt unregistered [ 4249.827296] Key type ._llcrypt unregistered [ 4300.656156] Key type ._llcrypt registered [ 4300.661091] Key type .llcrypt registered [ 4300.834924] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_hostid [ 4319.361832] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [ 4320.356041] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4320.556773] alg: No test for adler32 (adler32-zlib) [ 4321.880688] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 4322.189905] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [ 4323.935185] Key type lgssc registered [ 4325.395176] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4361.139811] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4361.182945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4362.504927] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4362.541363] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4362.658937] Lustre: lustre-MDT0000: new disk, initializing [ 4362.848345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4362.898404] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4368.719400] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 4380.940614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4381.156882] Lustre: 148911:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4381.276356] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4381.285076] Lustre: Skipped 1 previous similar message [ 4381.478884] Lustre: lustre-MDT0001: new disk, initializing [ 4381.715498] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4381.827885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4381.847508] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4387.430304] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 4397.197797] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4403.610889] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4410.494323] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4410.816272] Lustre: lustre-OST0000: new disk, initializing [ 4410.824290] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4410.832085] Lustre: 150883:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4410.919302] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4416.863683] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 4418.132574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4418.145777] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4418.256424] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4425.172328] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4429.227429] Lustre: Failing over lustre-OST0000 [ 4429.388491] Lustre: server umount lustre-OST0000 complete [ 4446.175145] Lustre: 147138:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788157952/real 1788157952] req@00000000559abdda x1875019399463168/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788157968 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4446.210767] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4446.978868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4447.321075] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4447.350219] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4447.365807] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4448.086653] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4449.247294] Lustre: 147139:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788157955/real 1788157955] req@00000000734a3b0c x1875019399464320/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788157971 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4449.272176] Lustre: 147139:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4453.227976] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 4454.368090] Lustre: 147138:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788157960/real 1788157960] req@0000000056445ba6 x1875019399465728/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788157976 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4454.396359] Lustre: 147138:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4456.510607] Lustre: lustre-OST0000: Recovery over after 0:08, of 3 clients 3 recovered and 0 were evicted. [ 4456.512481] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4457.022023] Lustre: Failing over lustre-MDT0000 [ 4457.481373] Lustre: server umount lustre-MDT0000 complete [ 4458.464382] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4458.470130] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4460.127554] Lustre: 147138:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788157966/real 1788157966] req@0000000040dd5468 x1875019399466752/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788157982 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4460.171359] Lustre: 147138:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4461.538065] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4461.572261] Lustre: Skipped 1 previous similar message [ 4463.405630] LustreError: 148920:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4463.429801] LustreError: 148920:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 4466.659698] LustreError: 148918:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4466.686931] LustreError: 148918:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4468.530879] LustreError: 148920:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4471.778033] LustreError: 150476:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4471.830049] LustreError: 150476:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4476.901611] LustreError: 148924:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4476.927756] LustreError: 148924:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4477.920171] Lustre: 147139:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788157983/real 1788157983] req@000000009f839c63 x1875019399476864/t0(0) o400->MGC192.168.204.139@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788157999 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4477.962424] Lustre: 147139:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4477.984234] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4478.711585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4484.908158] LustreError: 150477:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4484.927273] LustreError: 150477:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4487.516590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4487.557019] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4490.031510] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4492.637548] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 4492.788835] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4492.794986] Lustre: Skipped 1 previous similar message [ 4492.871066] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 4492.919084] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4503.016818] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4503.021821] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4503.037365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4503.069445] Lustre: Skipped 1 previous similar message [ 4508.129105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4508.150951] Lustre: Skipped 2 previous similar messages [ 4508.680902] Lustre: server umount lustre-OST0000 complete [ 4518.371297] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4518.382862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4518.395697] Lustre: Skipped 1 previous similar message [ 4518.414471] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4519.801926] Lustre: server umount lustre-MDT0000 complete [ 4523.488794] LustreError: 148923:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4523.513972] LustreError: 148923:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4524.544576] LustreError: 148905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158046 with bad export cookie 15645376862598151411 [ 4524.547192] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4524.557149] LustreError: 148905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4525.360511] Lustre: server umount lustre-MDT0001 complete [ 4537.534655] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [ 4540.535641] Key type lgssc unregistered [ 4540.935283] LNet: 154537:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4540.944261] LNetError: 154537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4540.960469] LNet: Removed LNI 192.168.204.139@tcp [ 4541.921336] Key type .llcrypt unregistered [ 4541.925874] Key type ._llcrypt unregistered [ 4581.301742] Key type ._llcrypt registered [ 4581.303696] Key type .llcrypt registered [ 4581.412415] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [ 4582.850873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4582.914291] alg: No test for adler32 (adler32-zlib) [ 4583.928296] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 4584.186671] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [ 4585.887191] Key type lgssc registered [ 4587.326178] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4595.846367] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4595.865281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4597.347171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4602.544824] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 4607.979958] LustreError: 156059:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4608.007898] LustreError: 156059:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4612.584189] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4613.096315] LustreError: 156060:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4616.885077] LustreError: 156088:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4616.912774] LustreError: 156088:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4616.925583] LustreError: 156088:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 4617.396506] Lustre: server umount lustre-MDT0000 complete [ 4633.516995] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_hostid [ 4645.733268] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [ 4687.625506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4687.942342] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4687.982174] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4688.056243] Lustre: lustre-MDT0000: new disk, initializing [ 4688.127166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4688.154786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4693.060382] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 4705.147327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4705.346491] Lustre: 159393:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4705.411953] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4705.418352] Lustre: Skipped 1 previous similar message [ 4705.550393] Lustre: lustre-MDT0001: new disk, initializing [ 4705.726019] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4705.791992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4705.805540] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4711.071604] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 4722.206468] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4730.084738] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4738.774969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4739.113960] Lustre: lustre-OST0000: new disk, initializing [ 4739.118284] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4739.127991] Lustre: 161363:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4739.284216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4744.761235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4744.774229] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4744.936643] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4746.812468] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 4756.392930] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4761.733715] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4763.916460] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 02:38:04 (1788158284) [ 5007.839682] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5007.851237] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5007.862303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5011.423917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5011.424270] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5013.962193] Lustre: server umount lustre-MDT0000 complete [ 5016.546433] LustreError: 161722:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5016.579415] LustreError: 161722:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5018.869067] LustreError: 161374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158540 with bad export cookie 4322887948773631702 [ 5018.871217] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5018.880528] LustreError: 161374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5021.664540] LustreError: 163139:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5021.664674] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5021.727398] Lustre: Skipped 1 previous similar message [ 5021.734244] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5021.745294] Lustre: Skipped 2 previous similar messages [ 5025.529624] Lustre: server umount lustre-MDT0001 complete [ 5035.900850] Lustre: server umount lustre-OST0000 complete [ 5047.179791] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_hostid [ 5058.770431] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [ 5105.108828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5105.362553] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5105.443683] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5105.586620] Lustre: lustre-MDT0000: new disk, initializing [ 5105.716585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5105.751636] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5110.696119] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 5121.468569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5121.643190] Lustre: 166031:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5121.698346] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5121.702246] Lustre: Skipped 1 previous similar message [ 5121.781394] Lustre: lustre-MDT0001: new disk, initializing [ 5121.879257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5121.928508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5121.940832] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5126.335290] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 5135.877470] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5141.050594] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5147.541186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5147.901843] Lustre: lustre-OST0000: new disk, initializing [ 5147.911383] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5147.924917] Lustre: 167997:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5148.010962] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5154.206514] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 5156.926167] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5156.940245] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5157.053267] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5163.552957] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5168.386140] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5168.689494] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5173.556892] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5173.956335] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5177.830783] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5177.832764] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5177.838507] LustreError: Skipped 1 previous similar message [ 5177.852632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5177.865032] Lustre: Skipped 2 previous similar messages [ 5182.554504] Lustre: server umount lustre-OST0000 complete [ 5188.064872] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5188.070890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5188.077963] Lustre: Skipped 2 previous similar messages [ 5192.495535] Lustre: server umount lustre-MDT0000 complete [ 5193.185044] LustreError: 167323:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5193.205641] LustreError: 167323:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5196.032782] LustreError: 166825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158717 with bad export cookie 4322887948773790539 [ 5196.036263] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5196.047412] LustreError: 166825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5196.300141] Lustre: server umount lustre-MDT0001 complete [ 5205.032455] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:45:25 (1788158725) [ 5221.525394] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_hostid [ 5230.996893] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [ 5265.138371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5265.489123] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5265.548039] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5265.633249] Lustre: lustre-MDT0000: new disk, initializing [ 5265.785753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5265.807286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5271.748742] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 5284.177698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5284.300834] Lustre: 172753:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5284.552069] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5288.720099] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 5298.765221] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5303.990372] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5310.181927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5310.472224] Lustre: lustre-OST0000: new disk, initializing [ 5310.481595] Lustre: Skipped 1 previous similar message [ 5310.487133] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5310.490861] Lustre: Skipped 2 previous similar messages [ 5310.496406] Lustre: 174727:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5317.048780] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 5317.733994] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5317.751211] Lustre: Skipped 1 previous similar message [ 5317.755736] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5317.864484] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5326.613728] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5333.475849] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5333.480872] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5333.492331] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5333.492340] Lustre: Skipped 1 previous similar message [ 5333.535671] Lustre: Skipped 1 previous similar message [ 5339.464559] Lustre: server umount lustre-OST0000 complete [ 5349.866451] Lustre: server umount lustre-MDT0000 complete [ 5353.813594] LustreError: 175836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158875 with bad export cookie 4322887948773791421 [ 5353.817324] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5353.828433] LustreError: 175836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5354.204372] Lustre: server umount lustre-MDT0001 complete [ 5362.451233] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:48:02 (1788158882) [ 5377.937607] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_hostid [ 5388.141031] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [ 5425.000953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5425.307672] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5425.347762] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5425.484971] Lustre: lustre-MDT0000: new disk, initializing [ 5425.543436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5425.547979] Lustre: Skipped 2 previous similar messages [ 5425.560880] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5430.445680] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 5443.333529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5443.497784] Lustre: 179055:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5443.761285] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5447.984925] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 5458.229853] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5464.835860] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5472.269755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5472.697490] Lustre: 181030:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5479.805370] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 5481.550473] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5481.750281] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5489.743749] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5502.434309] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5502.464085] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5502.485400] Lustre: Skipped 2 previous similar messages [ 5502.499461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5502.512932] Lustre: Skipped 8 previous similar messages [ 5505.317974] Lustre: server umount lustre-OST0000 complete [ 5510.625575] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5510.643130] LustreError: Skipped 1 previous similar message [ 5516.959411] Lustre: server umount lustre-MDT0000 complete [ 5517.808598] LustreError: 179061:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5517.836893] LustreError: 179061:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5522.216288] LustreError: 179047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159044 with bad export cookie 4322887948773792506 [ 5522.217344] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5522.230588] LustreError: 179047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5522.841100] Lustre: server umount lustre-MDT0001 complete [ 5536.649560] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [ 5539.649381] Key type lgssc unregistered [ 5539.931596] LNet: 182890:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5539.946922] LNetError: 182890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5539.960953] LNet: Removed LNI 192.168.204.139@tcp [ 5540.913957] Key type .llcrypt unregistered [ 5540.917071] Key type ._llcrypt unregistered [ 5565.209814] Key type ._llcrypt registered [ 5565.218903] Key type .llcrypt registered [ 5565.413816] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [ 5567.103814] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5567.190659] alg: No test for adler32 (adler32-zlib) [ 5568.230291] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 5568.447438] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [ 5570.104609] Key type lgssc registered [ 5571.449609] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5582.733523] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5582.764964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5584.434803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5590.528879] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 5594.594038] LustreError: 184307:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5594.610770] LustreError: 184307:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5599.101704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5599.559071] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5604.282730] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 5612.557521] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5618.995857] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5626.847708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5627.189745] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5629.246481] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5634.955667] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 5642.241469] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5665.253820] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5665.255508] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5665.261036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5665.265387] LustreError: Skipped 1 previous similar message [ 5665.302806] Lustre: Skipped 1 previous similar message [ 5667.515593] Lustre: server umount lustre-OST0000 complete [ 5671.394723] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5671.398831] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5671.408219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5671.416585] Lustre: Skipped 1 previous similar message [ 5675.490142] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5675.494943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5675.533320] Lustre: Skipped 1 previous similar message [ 5677.494502] Lustre: server umount lustre-MDT0000 complete [ 5680.619542] LustreError: 184302:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5680.634957] LustreError: 184302:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5681.612249] LustreError: 187121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159203 with bad export cookie 301848495095040170 [ 5681.613421] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5681.624090] LustreError: 187121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5682.050269] Lustre: server umount lustre-MDT0001 complete [ 5693.630418] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [ 5696.580737] Key type lgssc unregistered [ 5696.882928] LNet: 187873:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5696.891897] LNetError: 187873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5697.957192] LNet: Removed LNI 192.168.204.139@tcp [ 5698.891755] Key type .llcrypt unregistered [ 5698.894064] Key type ._llcrypt unregistered [ 5724.249267] Key type ._llcrypt registered [ 5724.251715] Key type .llcrypt registered [ 5724.396697] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [ 5725.782356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5725.948108] alg: No test for adler32 (adler32-zlib) [ 5727.051972] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 5727.406540] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [ 5729.119307] Key type lgssc registered [ 5730.298496] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5738.267972] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5738.311330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5740.018956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5744.840425] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 5749.225460] LustreError: 189292:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5749.242676] LustreError: 189292:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5754.352192] LustreError: 189291:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5754.566439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5755.135991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5760.082386] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 5767.680479] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5773.252431] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5780.378880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5780.878815] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5785.012348] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5787.308609] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 5794.970765] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5805.536853] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5805.549166] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5805.572153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5807.971283] Lustre: lustre-OST0000: Not available for connect from 192.168.204.39@tcp (stopping) [ 5809.636356] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5809.641638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5809.654055] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5809.679647] Lustre: Skipped 1 previous similar message [ 5813.046465] Lustre: lustre-OST0000: Not available for connect from 192.168.204.39@tcp (stopping) [ 5818.164196] Lustre: lustre-OST0000: Not available for connect from 192.168.204.39@tcp (stopping) [ 5818.180435] Lustre: Skipped 2 previous similar messages [ 5819.359736] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5819.522346] Lustre: server umount lustre-OST0000 complete [ 5826.836414] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9fcac47dc800.ost_server_uuid 50 [ 5828.375421] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5844.450710] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5844.453928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5844.473598] Lustre: Skipped 1 previous similar message [ 5844.487325] Lustre: Skipped 1 previous similar message [ 5848.803465] Lustre: server umount lustre-MDT0000 complete [ 5849.571035] LustreError: 189292:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5849.602979] LustreError: 189292:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5853.205744] LustreError: 191133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159375 with bad export cookie 6209061440193907709 [ 5853.207771] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5853.225420] LustreError: 191133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5853.532364] Lustre: server umount lustre-MDT0001 complete [ 5862.991685] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 02:56:23 (1788159383) [ 5869.837617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5870.256940] LustreError: 193206:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5870.342906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5875.279507] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 5875.693164] LustreError: 193207:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5883.553328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5883.917490] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5888.248612] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 5895.746461] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5901.341684] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5908.648334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5909.071869] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5915.249105] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5915.515983] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 5922.740961] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5929.983100] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5930.142511] Lustre: lustre-OST0001: new disk, initializing [ 5930.147236] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5930.152888] Lustre: 196045:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5930.231699] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5936.978701] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 5938.311480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5938.325110] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5938.424704] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5948.408577] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5951.456343] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5951.461667] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5951.487665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5965.279197] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5965.471846] Lustre: server umount lustre-OST0000 complete [ 5969.386819] LustreError: 195026:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5969.414028] LustreError: 195026:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5975.094592] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5975.365481] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5980.199356] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5980.442592] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5989.866519] LustreError: 195028:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5989.894266] LustreError: 195028:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 11 previous similar messages [ 6000.100248] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6000.118036] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6000.122237] LustreError: Skipped 1 previous similar message [ 6000.131962] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6000.137684] Lustre: Skipped 2 previous similar messages [ 6000.139454] Lustre: Skipped 9 previous similar messages [ 6001.291116] Lustre: server umount lustre-OST0001 complete [ 6006.753061] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6010.337583] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6010.360196] Lustre: Skipped 1 previous similar message [ 6012.033846] Lustre: server umount lustre-MDT0000 complete [ 6016.777848] LustreError: 193186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159538 with bad export cookie 6209061440193908787 [ 6016.787745] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6016.790425] LustreError: 193186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6017.608964] Lustre: server umount lustre-MDT0001 complete [ 6045.982510] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 02:59:26 (1788159566) [ 6053.956176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6054.314082] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6054.409260] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6054.816463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6059.029217] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 6068.570040] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6068.761225] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6068.823216] Lustre: 199492:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6073.812721] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 6081.183465] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6087.870689] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6095.130198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6095.362684] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6103.073710] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 6105.260915] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6110.824407] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6117.959928] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6118.017764] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6124.532034] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 6132.953902] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6136.289504] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6136.313153] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6136.334617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6136.339692] Lustre: Skipped 4 previous similar messages [ 6142.311808] Lustre: server umount lustre-OST0000 complete [ 6143.800664] LustreError: 201347:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6143.817150] LustreError: 201347:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 6160.358040] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6164.602144] Lustre: server umount lustre-OST0001 complete [ 6170.614257] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6170.638313] Lustre: Skipped 4 previous similar messages [ 6175.032870] Lustre: server umount lustre-MDT0000 complete [ 6178.374453] LustreError: 201365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159700 with bad export cookie 6209061440193910194 [ 6178.376767] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6178.388602] LustreError: 201365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6178.693740] Lustre: server umount lustre-MDT0001 complete [ 6200.820923] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 03:02:01 (1788159721) [ 6215.952994] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_hostid [ 6225.948777] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [ 6263.234087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6263.465779] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6263.508784] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6263.576991] Lustre: lustre-MDT0000: new disk, initializing [ 6263.649090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6263.656365] Lustre: Skipped 3 previous similar messages [ 6263.673275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6268.618288] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 6279.263759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6279.413988] Lustre: 207598:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6279.555773] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6279.563624] Lustre: Skipped 1 previous similar message [ 6279.649275] Lustre: lustre-MDT0001: new disk, initializing [ 6279.805698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6279.814663] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6284.132939] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 6292.722162] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6298.211665] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6304.502435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6304.820859] Lustre: lustre-OST0000: new disk, initializing [ 6304.827382] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6304.842882] Lustre: 209566:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6310.975864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6310.988849] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6311.086247] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6311.665965] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 6321.421820] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6326.287580] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6326.535578] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6331.196857] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6331.443640] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6337.005156] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6337.022215] LustreError: Skipped 1 previous similar message [ 6337.030317] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6337.055485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6337.060582] Lustre: Skipped 7 previous similar messages [ 6339.677954] Lustre: server umount lustre-OST0000 complete [ 6350.653356] Lustre: server umount lustre-MDT0000 complete [ 6352.357674] LustreError: 207609:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6352.396846] LustreError: 207609:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 6355.058685] LustreError: 207590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159876 with bad export cookie 6209061440193911755 [ 6355.059737] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6355.070275] LustreError: 207590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6355.702273] Lustre: server umount lustre-MDT0001 complete [ 6367.331725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6373.485657] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 6384.918901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6390.914661] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 6399.163347] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6404.954422] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6411.942719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6412.400867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6412.405876] Lustre: Skipped 4 previous similar messages [ 6419.714913] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 6429.752993] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6436.352254] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6436.669874] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6443.857236] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6444.104192] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6448.104161] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6448.116172] LustreError: Skipped 2 previous similar messages [ 6453.095935] Lustre: server umount lustre-OST0000 complete [ 6461.266568] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6467.293480] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6496.731407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6501.300363] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6501.323231] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6505.240375] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 6514.259259] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6515.841412] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6516.730803] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6516.761189] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6521.851655] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6521.867806] Lustre: Skipped 6 previous similar messages [ 6527.620983] Lustre: server umount lustre-OST0000 complete [ 6546.399416] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6546.663578] Lustre: server umount lustre-MDT0000 complete [ 6547.425251] LustreError: 212515:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6547.442694] LustreError: 212515:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 6551.061252] LustreError: 216593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160072 with bad export cookie 6209061440193912637 [ 6551.073529] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6551.075185] LustreError: 216593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6551.486750] Lustre: server umount lustre-MDT0001 complete [ 6559.668640] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 03:08:00 (1788160080) [ 6566.216879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6571.983960] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 6581.159514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6586.301806] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 6593.436355] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6599.976206] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6607.670155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6615.340457] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 6623.240366] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6628.243247] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6628.496950] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6632.976210] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6633.268285] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6640.285782] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6640.505809] Lustre: lustre-OST0001: new disk, initializing [ 6640.508763] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6640.514208] Lustre: 221124:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6648.100642] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 6650.472213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6650.498541] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6650.617261] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6657.704350] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6662.435513] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6662.669588] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6668.026729] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6668.320446] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6671.341117] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6671.356290] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6671.359436] LustreError: Skipped 5 previous similar messages [ 6671.377578] Lustre: Skipped 17 previous similar messages [ 6676.474828] Lustre: server umount lustre-OST0001 complete [ 6682.235609] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6682.456613] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6686.710207] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6686.972388] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6715.170174] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6715.422334] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6715.431548] Lustre: Skipped 5 previous similar messages [ 6720.496399] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6720.517881] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6720.525868] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6720.550949] Lustre: Skipped 1 previous similar message [ 6721.225178] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 6727.508844] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6728.912425] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6736.754052] Lustre: server umount lustre-OST0001 complete [ 6760.376575] Lustre: server umount lustre-OST0000 complete [ 6779.362685] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6779.945133] Lustre: server umount lustre-MDT0000 complete [ 6785.754971] LustreError: 221123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160307 with bad export cookie 6209061440193913603 [ 6785.786137] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6785.799223] LustreError: 221123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6800.352621] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6800.685172] Lustre: server umount lustre-MDT0001 complete [ 6827.053716] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 03:12:26 (1788160346) [ 6834.713316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6835.029715] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6835.107088] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6840.737248] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 6850.472045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6850.614781] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6850.689229] Lustre: 226369:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6856.527661] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 6866.567739] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6873.242603] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6881.635895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6882.081526] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6889.971727] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 6898.828189] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6908.445096] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6908.530759] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6915.282803] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 6921.987843] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6923.295754] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6927.400912] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6927.616803] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6932.492808] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6932.720260] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6938.512278] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9fcac5866000.ost_server_uuid 50 [ 6940.506127] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9fcac5866000.ost_server_uuid in IDLE state after 0 sec [ 6941.870935] Lustre: Permanently deactivating lustre-OST0001 [ 6941.912860] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6946.119485] Lustre: Permanently reactivating lustre-OST0001 [ 6946.172807] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6946.180365] Lustre: Skipped 2 previous similar messages [ 6951.943650] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6951.963842] LustreError: Skipped 5 previous similar messages [ 6951.969799] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6951.989884] Lustre: Skipped 10 previous similar messages [ 6956.929695] Lustre: server umount lustre-OST0001 complete [ 6958.054199] LustreError: 228218:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6958.071971] LustreError: 228218:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 38 previous similar messages [ 6966.832962] Lustre: server umount lustre-OST0000 complete [ 6977.734844] Lustre: server umount lustre-MDT0000 complete [ 6982.398751] LustreError: 226364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160504 with bad export cookie 6209061440193914891 [ 6982.403382] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6982.408845] LustreError: 226364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6982.798983] Lustre: server umount lustre-MDT0001 complete [ 7007.427670] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 03:15:27 (1788160527) [ 7015.640541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7015.935408] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7015.973217] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7015.983298] Lustre: Skipped 2 previous similar messages [ 7020.678867] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 7030.712200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7030.949434] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7031.033791] Lustre: 233072:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7037.027462] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 7046.096570] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7052.613386] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7061.494126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7061.749599] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7061.783584] Lustre: Permanently deactivating lustre-OST0000 [ 7061.812087] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7068.717632] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7069.790617] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 7078.220241] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7085.395542] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7085.438651] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7092.056692] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 7094.009795] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7100.889269] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7103.402839] Lustre: Permanently reactivating lustre-OST0000 [ 7103.421780] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7103.429812] Lustre: Skipped 2 previous similar messages [ 7109.226952] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7109.233245] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 41 seconds. I think it's dead, and I am evicting it. exp 0000000060d37a85, cur 1788160631 deadline 1788160620 last 1788160590 [ 7109.242329] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7109.248100] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7109.311279] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7122.326765] Lustre: server umount lustre-OST0001 complete [ 7132.906337] Lustre: server umount lustre-OST0000 complete [ 7143.243276] Lustre: server umount lustre-MDT0000 complete [ 7147.813471] LustreError: 233814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160669 with bad export cookie 6209061440193916564 [ 7147.815978] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7147.843056] LustreError: 233814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7148.380138] Lustre: server umount lustre-MDT0001 complete [ 7159.463231] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [ 7162.565596] Key type lgssc unregistered [ 7162.992076] LNet: 237953:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7163.001434] LNetError: 237953:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7163.035403] LNet: Removed LNI 192.168.204.139@tcp [ 7163.970144] Key type .llcrypt unregistered [ 7163.973947] Key type ._llcrypt unregistered [ 7189.661919] Key type ._llcrypt registered [ 7189.664120] Key type .llcrypt registered [ 7189.742029] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [ 7191.033839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7191.084773] alg: No test for adler32 (adler32-zlib) [ 7192.150606] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 7192.386407] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [ 7194.063192] Key type lgssc registered [ 7195.350674] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7206.271456] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7206.288339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7207.919677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7212.421627] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 7217.126248] LustreError: 239418:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7217.143949] LustreError: 239418:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7221.229755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7221.562578] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7221.645099] Lustre: 239407:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7221.669336] Lustre: Permanently deactivating lustre-MDT0001 [ 7221.702186] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7222.145918] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7222.173173] LustreError: 239449:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7222.190231] LustreError: 239449:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7222.212708] LustreError: 239449:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 1, failed: rc = -108 [ 7222.290980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7228.452589] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 7235.680227] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7241.443324] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7248.669881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7249.129815] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7252.219283] LustreError: 241261:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7252.242163] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7255.598443] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 7257.574294] LustreError: 241439:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7262.695223] LustreError: 241263:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7264.061155] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7267.823314] LustreError: 241263:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7267.846754] LustreError: 241263:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7267.859770] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7272.252202] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7272.707621] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7278.101798] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7278.109105] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7281.031509] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 7291.007646] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7294.920819] Lustre: Permanently reactivating lustre-MDT0001 [ 7294.952950] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7294.962637] Lustre: Skipped 2 previous similar messages [ 7300.431488] Lustre: lustre-MDT0001: Client 5efe396c-65e0-41dc-9b74-398c7df30328 (at 192.168.204.39@tcp) reconnecting [ 7306.359910] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7308.096981] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7313.911921] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7314.106194] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7319.302908] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7319.529482] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7322.235536] Lustre: Permanently deactivating lustre-MDT0001 [ 7322.249466] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7322.256918] Lustre: Skipped 2 previous similar messages [ 7328.374698] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7334.880260] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7334.888395] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7334.902471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7336.850749] Lustre: server umount lustre-MDT0000 complete [ 7338.474780] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7338.479560] LustreError: 239419:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7338.493400] Lustre: Skipped 2 previous similar messages [ 7338.531416] LustreError: 239419:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7340.606551] LustreError: 240158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160862 with bad export cookie 2789303587998932429 [ 7340.616346] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7340.629444] LustreError: 240158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7340.836844] Lustre: server umount lustre-MDT0001 complete [ 7347.222552] Lustre: server umount lustre-OST0000 complete [ 7359.262763] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 03:21:19 (1788160879) [ 7369.404202] Lustre: server umount lustre-OST0001 complete [ 7376.786111] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_hostid [ 7389.321077] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [ 7430.348861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7430.552170] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7430.562165] Lustre: Skipped 2 previous similar messages [ 7430.604791] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7430.655810] Lustre: lustre-MDT0000: new disk, initializing [ 7430.716307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7430.728038] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7435.260612] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 7446.447030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7446.633826] Lustre: 247679:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7446.724655] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7446.739092] Lustre: Skipped 1 previous similar message [ 7446.845388] Lustre: lustre-MDT0001: new disk, initializing [ 7446.973290] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7447.058922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7447.074406] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7453.345876] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 7464.076646] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7470.384606] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7477.373595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7477.648518] Lustre: lustre-OST0000: new disk, initializing [ 7477.653573] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7477.660075] Lustre: 249647:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7477.776670] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7483.889460] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 7484.500912] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7484.513784] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7484.627348] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7493.225861] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7495.096906] LustreError: 247685:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7504.409621] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7504.575266] Lustre: lustre-OST0001: new disk, initializing [ 7504.578543] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7504.585092] Lustre: 250761:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7505.184220] LustreError: 247685:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7505.237352] LustreError: 249655:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7511.644620] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 7514.224226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7514.258149] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7514.340611] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7515.248593] LustreError: 249655:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7521.777705] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7523.372679] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7529.962346] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7529.962809] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7529.987713] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7531.568394] Lustre: server umount lustre-OST0001 complete [ 7535.088042] LustreError: 249628:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7535.105696] LustreError: 249628:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7550.443463] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7550.453743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7550.469781] Lustre: Skipped 1 previous similar message [ 7550.473228] Lustre: Skipped 2 previous similar messages [ 7554.525690] Lustre: server umount lustre-OST0000 complete [ 7559.649979] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7559.665198] LustreError: Skipped 1 previous similar message [ 7559.668316] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7559.688335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7559.695808] Lustre: Skipped 1 previous similar message [ 7564.311531] Lustre: server umount lustre-MDT0000 complete [ 7568.001642] LustreError: 247671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161089 with bad export cookie 2789303587998935243 [ 7568.020284] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7568.023579] LustreError: 247671:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7574.564568] Lustre: server umount lustre-MDT0001 complete [ 7584.807091] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [ 7587.446545] Key type lgssc unregistered [ 7587.753583] LNet: 252792:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7587.761510] LNetError: 252792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7587.786883] LNet: Removed LNI 192.168.204.139@tcp [ 7588.784351] Key type .llcrypt unregistered [ 7588.786366] Key type ._llcrypt unregistered [ 7629.327546] Key type ._llcrypt registered [ 7629.333515] Key type .llcrypt registered [ 7629.436552] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [ 7630.468023] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7630.503741] alg: No test for adler32 (adler32-zlib) [ 7631.536762] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 7631.779827] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [ 7633.448536] Key type lgssc registered [ 7634.534970] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7641.733935] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7641.753492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7642.983177] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7643.003871] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7643.127968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7646.830178] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 7653.725116] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7653.802215] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7653.820925] Lustre: 254999:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7654.027219] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7658.131438] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 7664.924963] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7670.223433] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7676.249556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7676.437732] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7676.635477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7683.724062] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7683.763203] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 7692.761170] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7745.504809] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7745.516424] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7745.537737] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7750.119605] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7750.128881] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7750.149927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7751.647090] Lustre: server umount lustre-OST0000 complete [ 7756.417116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7771.880836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7772.256791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7775.339122] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7775.350813] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7779.809167] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 7787.929203] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7791.728090] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7791.743446] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7827.942289] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7827.942519] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7827.943876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7827.943882] Lustre: Skipped 1 previous similar message [ 7827.958777] Lustre: Skipped 1 previous similar message [ 7833.062637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7833.076218] Lustre: Skipped 1 previous similar message [ 7834.109127] Lustre: server umount lustre-OST0000 complete [ 7838.176513] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7838.179392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7838.193159] Lustre: Skipped 1 previous similar message [ 7838.210737] Lustre: Skipped 2 previous similar messages [ 7844.085108] Lustre: server umount lustre-MDT0000 complete [ 7847.874404] LustreError: 255743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161369 with bad export cookie 1988692157286268885 [ 7847.894970] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7848.220615] Lustre: server umount lustre-MDT0001 complete [ 7859.188935] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [ 7862.984857] Key type lgssc unregistered [ 7863.436929] LNet: 260426:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7863.459905] LNetError: 260426:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7863.486923] LNet: Removed LNI 192.168.204.139@tcp [ 7864.486107] Key type .llcrypt unregistered [ 7864.488190] Key type ._llcrypt unregistered [ 7892.440559] Key type ._llcrypt registered [ 7892.447572] Key type .llcrypt registered [ 7892.567109] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [ 7893.438760] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7893.477566] alg: No test for adler32 (adler32-zlib) [ 7894.520838] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 7894.695201] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [ 7896.335192] Key type lgssc registered [ 7897.793987] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7906.817839] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7906.831297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7908.344736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7914.162374] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 7918.569755] LustreError: 261845:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7918.587301] LustreError: 261845:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7923.684094] LustreError: 261844:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7924.667919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7925.084035] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7930.347071] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 7938.429344] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7944.745403] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7953.108759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7953.680359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7954.763406] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7960.844534] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 7969.451805] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7994.856546] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7994.857028] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7994.877394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7994.888187] Lustre: Skipped 1 previous similar message [ 7999.627435] Lustre: server umount lustre-OST0000 complete [ 8005.088728] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8005.116348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8005.120051] Lustre: Skipped 2 previous similar messages [ 8005.138266] Lustre: Skipped 1 previous similar message [ 8010.210960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8010.232042] Lustre: Skipped 1 previous similar message [ 8014.307055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8014.314087] Lustre: Skipped 1 previous similar message [ 8018.401220] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8018.599699] Lustre: server umount lustre-MDT0000 complete [ 8019.425841] LustreError: 261840:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8019.453413] LustreError: 261840:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8022.641563] LustreError: 261824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161544 with bad export cookie 3058138255686663766 [ 8022.646784] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8022.656182] LustreError: 261824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8023.095792] Lustre: server umount lustre-MDT0001 complete [ 8034.185842] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [ 8037.669043] Key type lgssc unregistered [ 8038.126848] LNet: 266154:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8038.134765] LNetError: 266154:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8038.151532] LNet: Removed LNI 192.168.204.139@tcp [ 8039.099884] Key type .llcrypt unregistered [ 8039.106362] Key type ._llcrypt unregistered [ 8059.567573] Key type ._llcrypt registered [ 8059.569571] Key type .llcrypt registered [ 8059.667117] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [ 8061.078015] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8061.184791] alg: No test for adler32 (adler32-zlib) [ 8062.258853] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 8062.532667] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [ 8064.223749] Key type lgssc registered [ 8065.457667] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8073.457968] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8073.478818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8075.124651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8080.415990] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 8084.456804] LustreError: 267376:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8084.483500] LustreError: 267376:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8089.312981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8089.575462] LustreError: 267375:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8089.928120] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8095.019636] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 8104.712021] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8112.155790] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8120.179410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8120.743893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8126.864666] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 8135.498765] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8144.308956] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8158.715631] Lustre: server umount lustre-OST0000 complete [ 8164.835964] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8164.855851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8164.866930] Lustre: Skipped 1 previous similar message [ 8169.956860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8169.969503] Lustre: Skipped 1 previous similar message [ 8173.115505] Lustre: server umount lustre-MDT0000 complete [ 8175.074855] LustreError: 268364:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8175.089315] LustreError: 268364:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8176.095136] Lustre: 266690:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788161682/real 1788161682] req@00000000cae898f6 x1875023321136512/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788161698 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8176.097103] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8176.121314] Lustre: 266690:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 8176.147026] Lustre: Skipped 1 previous similar message [ 8177.063503] LustreError: 267356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161698 with bad export cookie 5618817043056674849 [ 8177.074549] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8177.473180] Lustre: server umount lustre-MDT0001 complete [ 8187.657021] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [ 8190.416980] Key type lgssc unregistered [ 8190.687600] LNet: 271075:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8190.702408] LNetError: 271075:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8190.729642] LNet: Removed LNI 192.168.204.139@tcp [ 8191.849668] Key type .llcrypt unregistered [ 8191.853242] Key type ._llcrypt unregistered [ 8214.499781] Key type ._llcrypt registered [ 8214.502875] Key type .llcrypt registered [ 8214.610051] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [ 8216.047201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8216.085308] alg: No test for adler32 (adler32-zlib) [ 8217.251212] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 8217.592174] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [ 8219.359359] Key type lgssc registered [ 8221.205095] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8231.375211] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8231.425337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8232.926453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8237.834352] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 8242.149172] LustreError: 272494:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8242.182767] LustreError: 272494:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8247.281760] LustreError: 272495:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8248.934585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8249.724538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8255.222088] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 8263.187840] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8268.976784] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8275.969451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8276.406901] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8277.429557] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8283.616175] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 8291.909985] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8320.483249] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8320.497389] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8320.502418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8320.513460] Lustre: Skipped 1 previous similar message [ 8320.528519] Lustre: Skipped 1 previous similar message [ 8325.518055] Lustre: server umount lustre-OST0000 complete [ 8330.721719] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8330.731678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8330.739167] Lustre: Skipped 1 previous similar message [ 8335.404652] Lustre: server umount lustre-MDT0000 complete [ 8335.841885] LustreError: 272494:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8335.857388] LustreError: 272494:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8339.269996] LustreError: 272477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161861 with bad export cookie 15440442054167409139 [ 8339.278047] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8339.282037] LustreError: 272477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8339.696666] Lustre: server umount lustre-MDT0001 complete [ 8352.049461] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [ 8355.054148] Key type lgssc unregistered [ 8355.365143] LNet: 276801:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8355.386430] LNetError: 276801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8355.419309] LNet: Removed LNI 192.168.204.139@tcp [ 8356.548665] Key type .llcrypt unregistered [ 8356.551084] Key type ._llcrypt unregistered [ 8376.183940] Key type ._llcrypt registered [ 8376.187656] Key type .llcrypt registered [ 8376.325576] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [ 8377.543787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8377.557836] alg: No test for adler32 (adler32-zlib) [ 8378.597366] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 8378.814118] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [ 8380.495241] Key type lgssc registered [ 8381.954214] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8390.433261] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8390.473968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8391.900521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8396.705518] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 8401.385216] LustreError: 278039:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8401.414234] LustreError: 278039:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8405.617029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8406.060717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8410.773058] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 8418.283225] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8423.957668] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8431.235354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8431.908671] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8437.061937] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8438.956847] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 8446.447483] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8462.821937] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8462.829543] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8462.859305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8465.892577] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8465.897726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8467.419727] Lustre: server umount lustre-OST0000 complete [ 8472.545175] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8472.556543] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8472.582378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8472.589733] Lustre: Skipped 1 previous similar message [ 8476.129485] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8476.130390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8476.159260] Lustre: Skipped 1 previous similar message [ 8477.915669] Lustre: server umount lustre-MDT0000 complete [ 8481.252264] LustreError: 278024:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8481.304202] LustreError: 278024:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8482.405804] LustreError: 278005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788162004 with bad export cookie 11487422955934893890 [ 8482.406182] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8482.415870] LustreError: 278005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8482.834238] Lustre: server umount lustre-MDT0001 complete [ 8496.395862] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [ 8500.365758] Key type lgssc unregistered [ 8500.639121] LNet: 281741:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8500.649994] LNetError: 281741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8500.680897] LNet: Removed LNI 192.168.204.139@tcp [ 8501.596694] Key type .llcrypt unregistered [ 8501.602102] Key type ._llcrypt unregistered [ 8514.667898] Key type ._llcrypt registered [ 8514.669698] Key type .llcrypt registered [ 8514.815542] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8532.284377] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_hostid [ 8548.284886] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [ 8549.362037] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8549.535595] alg: No test for adler32 (adler32-zlib) [ 8550.640951] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 8550.912097] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [ 8552.575296] Key type lgssc registered [ 8553.822535] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8586.622148] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8586.642506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8587.895876] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8587.940495] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8588.054873] Lustre: lustre-MDT0000: new disk, initializing [ 8588.172539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8588.224941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8592.546783] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 8602.203526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8602.280310] Lustre: 285296:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8602.364825] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8602.376430] Lustre: Skipped 1 previous similar message [ 8602.501771] Lustre: lustre-MDT0001: new disk, initializing [ 8602.610731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8602.659702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8602.675116] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8606.958944] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 8616.565227] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8622.350806] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8629.296233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8629.551935] Lustre: lustre-OST0000: new disk, initializing [ 8629.555664] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8629.562559] Lustre: 287263:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8629.655823] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8635.009302] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 8639.041340] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8639.051581] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8639.173506] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8644.421020] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8649.684251] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8649.992920] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8655.340163] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8655.561365] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8659.936421] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8659.943089] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8659.954097] LustreError: Skipped 1 previous similar message [ 8659.954861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8659.998599] Lustre: Skipped 1 previous similar message [ 8663.710490] Lustre: server umount lustre-OST0000 complete [ 8669.155086] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8669.162821] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8669.183284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8669.186274] Lustre: Skipped 1 previous similar message [ 8670.177900] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8670.197806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8670.203306] Lustre: Skipped 1 previous similar message [ 8673.856261] Lustre: server umount lustre-MDT0000 complete [ 8677.995859] LustreError: 285288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788162199 with bad export cookie 13000560908657761137 [ 8678.005777] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8678.008877] LustreError: 285288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8678.586713] Lustre: server umount lustre-MDT0001 complete [ 8687.225859] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 03:43:27 (1788162207) [ 8694.944766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8695.482992] LustreError: 289674:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8695.504869] LustreError: 289674:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8695.595524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8700.412525] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 8700.897513] LustreError: 289675:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8706.046464] LustreError: 289674:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8711.150611] LustreError: 289675:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8715.480280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8716.290835] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8716.519928] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8723.564791] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 8731.705589] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8737.340780] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8744.634626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8745.043946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8751.200897] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 8758.990863] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8776.176300] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8776.177843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8776.178157] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8776.197253] Lustre: Skipped 1 previous similar message [ 8776.234578] Lustre: Skipped 1 previous similar message [ 8781.342800] Lustre: server umount lustre-OST0000 complete [ 8786.402197] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8786.425441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8786.430149] Lustre: Skipped 1 previous similar message [ 8791.520582] LustreError: 289674:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8791.539082] LustreError: 289674:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8791.601938] Lustre: server umount lustre-MDT0000 complete [ 8795.292467] LustreError: 292178:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788162317 with bad export cookie 13000560908657762012 [ 8795.306532] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8795.628982] Lustre: server umount lustre-MDT0001 complete [ 8804.612853] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [ 8807.641541] Key type lgssc unregistered [ 8807.878871] LNet: 293231:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8807.887798] LNetError: 293231:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8807.908625] LNet: Removed LNI 192.168.204.139@tcp [ 8808.565330] Key type .llcrypt unregistered [ 8808.567361] Key type ._llcrypt unregistered [ 8838.415571] Key type ._llcrypt registered [ 8838.417090] Key type .llcrypt registered [ 8838.523121] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_hostid [ 8853.417902] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [ 8854.382375] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8854.419228] alg: No test for adler32 (adler32-zlib) [ 8855.486661] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 8855.718028] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [ 8857.464650] Key type lgssc registered [ 8858.968348] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8913.867804] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8913.946763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8915.156296] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8915.198555] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8915.350340] Lustre: lustre-MDT0000: new disk, initializing [ 8915.433065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8915.453356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8919.685255] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 8930.225961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8930.306451] Lustre: 297062:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8930.362078] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8930.368381] Lustre: Skipped 1 previous similar message [ 8930.446326] Lustre: lustre-MDT0001: new disk, initializing [ 8930.556668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8930.614505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8930.633556] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8934.693781] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 8944.271606] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8950.948541] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8958.024242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8958.386764] Lustre: lustre-OST2710: new disk, initializing [ 8958.391435] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8958.403675] Lustre: 299031:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8958.501059] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8964.700481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8964.720592] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8964.882367] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8964.928775] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 8974.881425] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8982.297812] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8982.533440] Lustre: lustre-OST03e8: new disk, initializing [ 8982.536922] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8982.554911] Lustre: 300093:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8982.702390] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8989.920231] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 8990.276239] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8990.285320] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8990.365778] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8997.146787] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9004.577175] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9005.003667] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9010.617669] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9010.974951] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9016.369878] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9016.801898] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9022.619503] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9023.039299] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9031.654665] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9031.661144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9031.667370] Lustre: Skipped 1 previous similar message [ 9031.688921] Lustre: Skipped 3 previous similar messages [ 9035.294954] Lustre: server umount lustre-MDT0000 complete [ 9038.993292] LustreError: 297053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788162560 with bad export cookie 15356733199523754083 [ 9039.008566] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9039.009914] LustreError: 297053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9039.410473] Lustre: server umount lustre-MDT0001 complete [ 9054.022508] Lustre: server umount lustre-OST2710 complete [ 9057.252754] Lustre: 294857:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788162563/real 1788162563] req@000000002f653dd0 x1875024152696192/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788162579 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9057.302575] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9057.339941] Lustre: Skipped 2 previous similar messages [ 9058.639735] Lustre: server umount lustre-OST03e8 complete [ 9079.823131] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_hostid [ 9090.376963] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [ 9123.040893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9123.250646] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9123.276553] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9123.363315] Lustre: lustre-MDT0000: new disk, initializing [ 9123.429116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9123.449064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9127.714641] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 9137.647683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9137.798171] Lustre: 305344:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9137.909840] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9137.918150] Lustre: Skipped 1 previous similar message [ 9138.062020] Lustre: lustre-MDT0001: new disk, initializing [ 9138.179368] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9138.226508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9138.249160] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9142.339516] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 9150.657511] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9155.911348] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9162.301422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9162.574372] Lustre: lustre-OST0000: new disk, initializing [ 9162.577897] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9162.585547] Lustre: 307312:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9162.668490] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9169.947416] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 9172.543961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9172.554501] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9172.643117] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9180.021243] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9185.286845] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9185.475730] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9190.807865] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9191.107551] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9193.441230] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9193.454191] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9193.465602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9193.472240] Lustre: Skipped 1 previous similar message [ 9198.560661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9198.570619] Lustre: Skipped 1 previous similar message [ 9199.359862] Lustre: server umount lustre-OST0000 complete [ 9204.722341] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9204.729360] LustreError: Skipped 1 previous similar message [ 9204.740316] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9204.762940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9204.772551] Lustre: Skipped 1 previous similar message [ 9208.803320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9208.803415] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9210.441899] Lustre: server umount lustre-MDT0000 complete [ 9213.921300] LustreError: 307771:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9213.973054] LustreError: 307771:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 9214.297429] LustreError: 305336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788162736 with bad export cookie 15356733199523755777 [ 9214.300432] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9214.317163] LustreError: 305336:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9214.582870] Lustre: server umount lustre-MDT0001 complete [ 9223.542661] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 03:52:23 (1788162743) [ 9225.221684] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9226.929269] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 03:52:27 (1788162747) [ 9230.310895] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [ 9258.830583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9259.111764] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9259.134982] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9259.400336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9264.419425] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 9274.405706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9274.554920] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9274.587315] Lustre: 311170:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9282.825504] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 9305.322682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9305.518517] Lustre: Denying initial registration attempt from nid 192.168.204.139@tcp, specified as failover [ 9305.524254] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9305.533806] LustreError: 312623:0:(tgt_mount.c:2688:server_fill_super()) Unable to start targets: -99 [ 9305.540087] LustreError: 312623:0:(tgt_mount.c:2206:server_put_super()) no obd lustre-OST0000 [ 9305.546996] LustreError: 312623:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9305.593459] Lustre: server umount lustre-OST0000 complete [ 9305.599350] LustreError: 312623:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9314.790656] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9314.801472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9314.806040] Lustre: Skipped 2 previous similar messages [ 9318.668959] Lustre: server umount lustre-MDT0000 complete [ 9319.906972] LustreError: 311176:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9319.937538] LustreError: 311176:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9322.061951] LustreError: 311160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788162843 with bad export cookie 15356733199523756659 [ 9322.069320] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9322.073746] LustreError: 311160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9322.320484] Lustre: server umount lustre-MDT0001 complete [ 9331.675360] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [ 9334.167887] Key type lgssc unregistered [ 9334.454857] LNet: 313744:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9334.466174] LNetError: 313744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9335.527585] LNet: Removed LNI 192.168.204.139@tcp [ 9336.332087] Key type .llcrypt unregistered [ 9336.333847] Key type ._llcrypt unregistered [ 9347.300481] Key type ._llcrypt registered [ 9347.303224] Key type .llcrypt registered [ 9347.420599] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [ 9348.540281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9348.685888] alg: No test for adler32 (adler32-zlib) [ 9349.904286] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 9350.146219] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [ 9351.801330] Key type lgssc registered [ 9353.255140] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9408.728716] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [ 9418.108774] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9418.140828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9419.505899] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9419.538554] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9419.747816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9424.466503] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 9433.907259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9434.011071] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9434.041154] Lustre: 316511:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9434.149179] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9434.394305] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9439.885437] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 9447.487166] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9452.778539] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9472.891274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9473.136629] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9473.298052] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9479.211202] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 9486.495917] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9492.961478] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9492.974377] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9492.989794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9493.483173] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9493.493097] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9494.762809] Lustre: server umount lustre-OST0000 complete [ 9501.154507] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9501.175351] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9501.224531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9501.238035] Lustre: Skipped 2 previous similar messages [ 9503.712890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9503.716387] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9505.661667] Lustre: server umount lustre-MDT0000 complete [ 9510.660100] LustreError: 318377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788163032 with bad export cookie 11716867036011055784 [ 9510.662733] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9510.685616] LustreError: 318377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9511.086655] Lustre: server umount lustre-MDT0001 complete [ 9522.886993] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [ 9525.801635] Key type lgssc unregistered [ 9526.092986] LNet: 320087:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9526.103805] LNetError: 320087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9526.121808] LNet: Removed LNI 192.168.204.139@tcp [ 9527.205655] Key type .llcrypt unregistered [ 9527.212771] Key type ._llcrypt unregistered [ 9555.213810] Key type ._llcrypt registered [ 9555.216843] Key type .llcrypt registered [ 9555.310764] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [ 9555.655324] Key type .llcrypt unregistered [ 9555.658541] Key type ._llcrypt unregistered [ 9571.552919] Key type ._llcrypt registered [ 9571.557777] Key type .llcrypt registered [ 9571.678968] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [ 9572.931676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9573.012867] alg: No test for adler32 (adler32-zlib) [ 9574.228458] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 9574.480469] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [ 9576.143228] Key type lgssc registered [ 9577.607945] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9611.503621] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing lnet_if_list [ 9613.004229] LNet: Added LNI 192.168.204.139@tcp1 [8/256/0/180] [ 9626.615888] loop0: detected capacity change from 0 to 204800000 [ 9626.904040] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9626.920139] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9626.929141] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9646.315381] loop0: detected capacity change from 0 to 204800000 [ 9646.588855] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9646.627135] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9646.641268] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9665.298822] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9665.331806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9666.637831] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9666.665716] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9666.918757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9671.969678] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 9682.116236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9682.236329] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9682.261978] Lustre: 324012:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9682.651148] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9688.764457] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 9698.265462] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9704.720498] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9709.892915] loop0: detected capacity change from 0 to 204800000 [ 9729.532299] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9729.852328] Lustre: sv28701-OST0000: new disk, initializing [ 9729.859609] Lustre: srv-sv28701-OST0000: No data found on store. Initialize space. [ 9729.869132] Lustre: 326093:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9729.943054] Lustre: sv28701-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9736.047807] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 9746.194709] loop1: detected capacity change from 0 to 204800000 [ 9764.713530] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9764.891516] Lustre: sv28701-OST0001: new disk, initializing [ 9764.898740] Lustre: srv-sv28701-OST0001: No data found on store. Initialize space. [ 9764.913908] Lustre: 327224:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9764.983138] Lustre: sv28701-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9771.702949] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 9786.773827] Lustre: server umount sv28701-OST0001 complete [ 9797.756158] Lustre: server umount sv28701-OST0000 complete [ 9810.921052] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9810.935235] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9810.972517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9813.474108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9813.476105] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9818.595030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9818.597379] Lustre: Skipped 1 previous similar message [ 9823.714895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9823.726284] Lustre: Skipped 1 previous similar message [ 9825.247305] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9825.528055] Lustre: server umount lustre-MDT0000 complete [ 9829.750158] LustreError: 324756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788163351 with bad export cookie 11723919048020798096 [ 9829.750882] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9829.758196] LustreError: 324756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9829.960280] Lustre: server umount lustre-MDT0001 complete [ 9842.952236] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [ 9845.954344] Key type lgssc unregistered [ 9846.262455] LNet: 329979:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9846.275542] LNetError: 329979:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9846.284229] LNet: Removed LNI 192.168.204.139@tcp [ 9847.334176] LNet: Removed LNI 192.168.204.139@tcp1 [ 9848.354131] Key type .llcrypt unregistered [ 9848.364897] Key type ._llcrypt unregistered [ 9867.643665] Key type ._llcrypt registered [ 9867.645796] Key type .llcrypt registered [ 9867.730524] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [ 9868.029416] Key type .llcrypt unregistered [ 9868.032376] Key type ._llcrypt unregistered [ 9884.119600] Key type ._llcrypt registered [ 9884.132995] Key type .llcrypt registered [ 9884.260993] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [ 9885.551159] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9885.588507] alg: No test for adler32 (adler32-zlib) [ 9886.693041] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 9886.926109] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [ 9888.641293] Key type lgssc registered [ 9889.702697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9912.509499] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing lnet_if_list [ 9913.749672] LNet: Added LNI 192.168.204.139@tcp1 [8/256/0/180] [ 9924.430821] loop0: detected capacity change from 0 to 204800000 [ 9924.674185] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9924.701031] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9924.725650] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9943.059763] loop0: detected capacity change from 0 to 204800000 [ 9943.350940] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9943.362841] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9943.383714] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9961.976343] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9962.002710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9963.294861] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9963.344712] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9963.498536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9967.712505] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 9976.695631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9976.903625] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9976.939159] Lustre: 333800:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9977.251553] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9981.812780] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [ 9990.402603] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9996.875773] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10003.112351] loop0: detected capacity change from 0 to 204800000 [10022.700838] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10023.219953] Lustre: sv12606-OST0000: new disk, initializing [10023.224626] Lustre: srv-sv12606-OST0000: No data found on store. Initialize space. [10023.237285] Lustre: 335880:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10023.340177] Lustre: sv12606-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10030.352187] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [10040.816461] loop1: detected capacity change from 0 to 204800000 [10059.889505] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10060.189303] Lustre: sv12606-OST0001: new disk, initializing [10060.203104] Lustre: srv-sv12606-OST0001: No data found on store. Initialize space. [10060.210053] Lustre: 336997:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10060.305994] Lustre: sv12606-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10068.383081] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [10082.929262] Lustre: server umount sv12606-OST0001 complete [10094.601215] Lustre: server umount sv12606-OST0000 complete [10107.872603] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10107.877086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10107.890397] Lustre: Skipped 1 previous similar message [10107.900604] Lustre: Skipped 1 previous similar message [10112.993058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10113.004173] Lustre: Skipped 1 previous similar message [10113.280381] Lustre: server umount lustre-MDT0000 complete [10117.795740] LustreError: 333794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788163639 with bad export cookie 2449148728624216192 [10117.801445] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10117.812988] LustreError: 333794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10118.182796] Lustre: server umount lustre-MDT0001 complete [10133.734848] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [10136.613869] Key type lgssc unregistered [10136.866815] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.39@tcp [10136.918818] LNet: 339753:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10136.926929] LNetError: 339753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10136.935507] LNet: Removed LNI 192.168.204.139@tcp [10137.961197] LNet: Removed LNI 192.168.204.139@tcp1 [10138.741950] Key type .llcrypt unregistered [10138.745781] Key type ._llcrypt unregistered [10156.615259] Key type ._llcrypt registered [10156.617320] Key type .llcrypt registered [10156.718241] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [10157.051612] Key type .llcrypt unregistered [10157.057629] Key type ._llcrypt unregistered [10172.296279] Key type ._llcrypt registered [10172.299353] Key type .llcrypt registered [10172.407830] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [10173.748634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10173.792054] alg: No test for adler32 (adler32-zlib) [10174.892441] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [10175.166303] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [10176.840560] Key type lgssc registered [10178.057320] Lustre: Echo OBD driver; http://www.lustre.org/ [10198.854337] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing lnet_if_list [10200.174875] LNet: Added LNI 192.168.204.139@tcp1 [8/256/0/180] [10210.340493] loop0: detected capacity change from 0 to 204800000 [10210.586059] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10210.600546] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10210.614725] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10227.897800] loop0: detected capacity change from 0 to 204800000 [10228.152885] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10228.177968] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10228.200791] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10246.378917] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10246.412230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10247.773658] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10247.798115] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10247.999416] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10252.661738] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [10262.245033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10262.404420] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10262.491727] Lustre: 343574:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10262.764404] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10267.648307] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [10276.517882] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10283.865919] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10289.341991] loop0: detected capacity change from 0 to 204800000 [10308.284263] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10308.729311] Lustre: sv31721-OST0000: new disk, initializing [10308.732471] Lustre: srv-sv31721-OST0000: No data found on store. Initialize space. [10308.737411] Lustre: 345654:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10308.799305] Lustre: sv31721-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10315.581215] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [10333.162928] Lustre: server umount sv31721-OST0000 complete [10348.530769] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10348.558673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10351.376306] Lustre: server umount lustre-MDT0000 complete [10355.139962] LustreError: 343569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788163877 with bad export cookie 10735862170326154091 [10355.150126] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10355.158563] LustreError: 343569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10355.534132] Lustre: server umount lustre-MDT0001 complete [10367.371153] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [10370.992535] Key type lgssc unregistered [10371.521808] LNet: 348105:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10371.535643] LNetError: 348105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10371.559241] LNet: Removed LNI 192.168.204.139@tcp [10372.584775] LNet: Removed LNI 192.168.204.139@tcp1 [10373.778375] Key type .llcrypt unregistered [10373.782916] Key type ._llcrypt unregistered [10395.355841] Key type ._llcrypt registered [10395.359487] Key type .llcrypt registered [10395.503145] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [10395.906556] Key type .llcrypt unregistered [10395.909728] Key type ._llcrypt unregistered [10412.598908] Key type ._llcrypt registered [10412.601046] Key type .llcrypt registered [10412.687799] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [10413.693895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10413.757909] alg: No test for adler32 (adler32-zlib) [10414.918623] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [10415.217402] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [10416.903177] Key type lgssc registered [10418.295763] Lustre: Echo OBD driver; http://www.lustre.org/ [10441.044265] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing lnet_if_list [10442.267893] LNet: Added LNI 192.168.204.139@tcp1 [8/256/0/180] [10452.086478] loop0: detected capacity change from 0 to 204800000 [10452.310460] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10452.322179] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10452.333759] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10471.205223] loop0: detected capacity change from 0 to 204800000 [10471.578996] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10471.606257] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10471.622799] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10489.839804] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10489.867315] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10491.126487] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10491.144541] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10491.353788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10496.101933] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [10504.840785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10505.079841] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10505.126662] Lustre: 351975:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10505.440107] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10509.974509] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [10517.836928] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10523.992967] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10528.686711] loop0: detected capacity change from 0 to 204800000 [10548.335560] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10548.764529] Lustre: sv5122-OST0000: new disk, initializing [10548.793920] Lustre: srv-sv5122-OST0000: No data found on store. Initialize space. [10548.803429] Lustre: 354052:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10548.903245] Lustre: sv5122-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10556.361547] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [10575.130958] Lustre: server umount sv5122-OST0000 complete [10590.177189] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10590.186317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10590.220029] Lustre: Skipped 1 previous similar message [10595.224453] Lustre: server umount lustre-MDT0000 complete [10599.582882] LustreError: 355755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788164121 with bad export cookie 9038616758762392104 [10599.599305] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10599.601038] LustreError: 355755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10600.132261] Lustre: server umount lustre-MDT0001 complete [10612.022812] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [10614.884397] Key type lgssc unregistered [10615.263586] LNet: 356506:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10615.272643] LNetError: 356506:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10615.285190] LNet: Removed LNI 192.168.204.139@tcp [10616.358393] LNet: Removed LNI 192.168.204.139@tcp1 [10617.358806] Key type .llcrypt unregistered [10617.362578] Key type ._llcrypt unregistered [10636.527219] Key type ._llcrypt registered [10636.531200] Key type .llcrypt registered [10636.632388] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [10636.859283] Key type .llcrypt unregistered [10636.862286] Key type ._llcrypt unregistered [10652.848697] Key type ._llcrypt registered [10652.850847] Key type .llcrypt registered [10652.952590] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [10654.084843] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10654.104418] alg: No test for adler32 (adler32-zlib) [10655.200907] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [10655.424041] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [10657.121727] Key type lgssc registered [10658.371743] Lustre: Echo OBD driver; http://www.lustre.org/ [10678.191322] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing lnet_if_list [10679.268069] LNet: Added LNI 192.168.204.139@tcp1 [8/256/0/180] [10687.794929] loop0: detected capacity change from 0 to 204800000 [10687.953506] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10687.966319] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10687.975223] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10704.629596] loop0: detected capacity change from 0 to 204800000 [10704.851057] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10704.870111] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10704.885067] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10722.695928] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10722.709340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10723.897695] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10723.913220] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10724.029501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10727.891903] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [10735.192771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10735.324128] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10735.369707] Lustre: 360328:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10735.683600] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10739.969920] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [10747.614378] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10752.900479] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10757.762988] loop0: detected capacity change from 0 to 204800000 [10776.063610] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10776.461906] Lustre: sv23490-OST0000: new disk, initializing [10776.473520] Lustre: srv-sv23490-OST0000: No data found on store. Initialize space. [10776.481997] Lustre: 362406:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10776.529617] Lustre: sv23490-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10781.949660] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [10791.842581] loop1: detected capacity change from 0 to 204800000 [10809.708931] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10809.844874] Lustre: sv23490-OST0001: new disk, initializing [10809.849479] Lustre: srv-sv23490-OST0001: No data found on store. Initialize space. [10809.856213] Lustre: 363575:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10809.911622] Lustre: sv23490-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10814.717945] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [10828.520764] Lustre: server umount sv23490-OST0001 complete [10839.554146] Lustre: server umount sv23490-OST0000 complete [10853.347261] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10853.358975] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10853.398800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10856.930276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10856.931877] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10858.500104] Lustre: server umount lustre-MDT0000 complete [10862.485228] LustreError: 360319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788164384 with bad export cookie 1944294859872095100 [10862.494788] LustreError: 360319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10862.498748] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10862.862289] Lustre: server umount lustre-MDT0001 complete [10875.649936] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [10879.010869] Key type lgssc unregistered [10879.330168] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.39@tcp [10879.374732] LNet: 366330:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10879.381290] LNetError: 366330:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10879.387733] LNet: Removed LNI 192.168.204.139@tcp [10880.422641] LNet: Removed LNI 192.168.204.139@tcp1 [10882.028675] Key type .llcrypt unregistered [10882.032818] Key type ._llcrypt unregistered [10904.609839] Key type ._llcrypt registered [10904.618946] Key type .llcrypt registered [10904.784438] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [10905.156989] Key type .llcrypt unregistered [10905.159900] Key type ._llcrypt unregistered [10923.037369] Key type ._llcrypt registered [10923.041624] Key type .llcrypt registered [10923.201811] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [10924.417378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10924.475056] alg: No test for adler32 (adler32-zlib) [10925.702699] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [10925.927847] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [10927.607273] Key type lgssc registered [10928.821550] Lustre: Echo OBD driver; http://www.lustre.org/ [10954.339547] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing lnet_if_list [10956.248989] LNet: Added LNI 192.168.204.139@tcp1 [8/256/0/180] [10988.833269] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [10991.664863] Key type lgssc unregistered [10991.941075] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.39@tcp [10991.982701] LNet: 370292:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10991.992864] LNetError: 370292:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10991.999876] LNet: Removed LNI 192.168.204.139@tcp [10992.014887] LNet: Removed LNI 192.168.204.139@tcp1 [10992.800807] Key type .llcrypt unregistered [10992.809963] Key type ._llcrypt unregistered [11012.671651] Key type ._llcrypt registered [11012.674256] Key type .llcrypt registered [11012.805947] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [11013.068758] Key type .llcrypt unregistered [11013.075893] Key type ._llcrypt unregistered [11034.273074] Key type ._llcrypt registered [11034.274703] Key type .llcrypt registered [11034.509384] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [11036.102965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11036.191846] alg: No test for adler32 (adler32-zlib) [11037.453347] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [11037.767433] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [11039.463193] Key type lgssc registered [11041.102142] Lustre: Echo OBD driver; http://www.lustre.org/ [11063.643040] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing lnet_if_list [11065.111558] LNet: Added LNI 192.168.204.139@tcp1 [8/256/0/180] [11066.732819] LNet: Added LNI 192.168.204.139@tcp2 [8/256/0/180] [11078.381607] loop0: detected capacity change from 0 to 204800000 [11078.618401] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11078.633607] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11078.657831] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11096.106480] loop0: detected capacity change from 0 to 204800000 [11096.253823] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11096.259903] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11096.266717] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11115.693859] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11115.736747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11116.980028] LNetError: 371621:0:(peer.c:2151:lnet_peer_add_nid()) Peer 192.168.204.139@tcp NID 192.168.204.139@tcp is already locked with peer 192.168.204.139@tcp [11117.229632] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11117.279867] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11117.537580] LNetError: 371621:0:(peer.c:2151:lnet_peer_add_nid()) Peer 192.168.204.139@tcp NID 192.168.204.139@tcp is already locked with peer 192.168.204.139@tcp [11117.567852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11124.466656] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [11134.068175] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11134.225372] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11134.268899] Lustre: 374209:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11134.478929] LNetError: 371621:0:(peer.c:2151:lnet_peer_add_nid()) Peer 192.168.204.139@tcp NID 192.168.204.139@tcp is already locked with peer 192.168.204.139@tcp [11134.673799] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11139.908553] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [11142.166976] LNetError: 371621:0:(peer.c:2151:lnet_peer_add_nid()) Peer 192.168.204.139@tcp NID 192.168.204.139@tcp is already locked with peer 192.168.204.139@tcp [11142.187587] LNetError: 371621:0:(peer.c:2151:lnet_peer_add_nid()) Skipped 3 previous similar messages [11149.453784] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11155.396330] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11161.169505] loop0: detected capacity change from 0 to 204800000 [11179.496616] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11179.769911] Lustre: sv18072-OST0000: new disk, initializing [11179.773875] Lustre: srv-sv18072-OST0000: No data found on store. Initialize space. [11179.789329] Lustre: 376289:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11179.921380] Lustre: sv18072-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11186.202507] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [11196.616567] loop1: detected capacity change from 0 to 204800000 [11215.664242] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11215.876056] Lustre: sv18072-OST0001: new disk, initializing [11215.882432] Lustre: srv-sv18072-OST0001: No data found on store. Initialize space. [11215.888206] Lustre: 377457:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11215.934468] Lustre: sv18072-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11222.555270] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [11235.075525] Lustre: server umount sv18072-OST0001 complete [11244.998486] Lustre: server umount sv18072-OST0000 complete [11257.828429] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11257.849667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11257.866418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11259.892986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11259.900075] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11262.914701] Lustre: server umount lustre-MDT0000 complete [11267.417717] LustreError: 374951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788164789 with bad export cookie 6901387842191037940 [11267.421888] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11267.426537] LustreError: 374951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11267.773754] Lustre: server umount lustre-MDT0001 complete [11279.654526] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [11282.954341] Key type lgssc unregistered [11283.281263] LNet: 380212:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11283.295844] LNetError: 380212:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11283.303316] LNet: Removed LNI 192.168.204.139@tcp [11284.319273] LNet: Removed LNI 192.168.204.139@tcp1 [11284.323742] LNetError: 380212:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11284.338644] LNetError: 380212:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [11284.372108] LNet: Removed LNI 192.168.204.139@tcp2 [11285.382847] Key type .llcrypt unregistered [11285.385489] Key type ._llcrypt unregistered [11306.998853] Key type ._llcrypt registered [11307.001330] Key type .llcrypt registered [11307.112591] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [11307.426804] Key type .llcrypt unregistered [11307.432750] Key type ._llcrypt unregistered [11324.220991] Key type ._llcrypt registered [11324.224038] Key type .llcrypt registered [11324.368172] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [11325.517641] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11325.621855] alg: No test for adler32 (adler32-zlib) [11326.664449] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [11326.847596] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [11328.487220] Key type lgssc registered [11329.837028] Lustre: Echo OBD driver; http://www.lustre.org/ [11351.342354] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing lnet_if_list [11352.477072] LNet: Added LNI 192.168.204.139@tcp1 [8/256/0/180] [11362.438445] loop0: detected capacity change from 0 to 204800000 [11362.623549] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11362.646130] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11362.657209] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11381.582939] loop0: detected capacity change from 0 to 204800000 [11381.769253] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11381.786676] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11381.794259] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11400.405911] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11400.440384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11401.851489] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11401.868587] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11402.092395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11406.112769] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [11414.801733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11414.875217] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11414.899224] Lustre: 384085:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11415.110486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11419.255368] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [11426.798089] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11432.570378] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11439.595617] loop0: detected capacity change from 0 to 204800000 [11459.628574] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11459.889609] Lustre: sv3997-OST0000: new disk, initializing [11459.896195] Lustre: srv-sv3997-OST0000: No data found on store. Initialize space. [11459.903604] Lustre: 386163:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11459.979150] Lustre: sv3997-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11466.209661] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [11477.351346] loop1: detected capacity change from 0 to 204800000 [11496.442212] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11496.827476] Lustre: sv3997-OST0001: new disk, initializing [11496.836948] Lustre: srv-sv3997-OST0001: No data found on store. Initialize space. [11496.849663] Lustre: 387280:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11496.984691] Lustre: sv3997-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11504.231564] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [11517.068755] Lustre: server umount sv3997-OST0001 complete [11526.217304] Lustre: server umount sv3997-OST0000 complete [11537.889654] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11537.898901] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11537.911068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11539.427258] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11539.433815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11539.444315] Lustre: Skipped 1 previous similar message [11544.546223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11544.554184] Lustre: Skipped 1 previous similar message [11549.667036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11549.673184] Lustre: Skipped 1 previous similar message [11550.175128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11550.339763] Lustre: server umount lustre-MDT0000 complete [11553.826602] LustreError: 384077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788165075 with bad export cookie 1115689205975550016 [11553.831834] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11553.844834] LustreError: 384077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11554.188949] Lustre: server umount lustre-MDT0001 complete [11564.958856] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [11567.894378] Key type lgssc unregistered [11568.188411] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.39@tcp [11568.226488] LNet: 390036:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11568.230835] LNetError: 390036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11569.247177] LNet: Removed LNI 192.168.204.139@tcp [11569.258546] LNetError: 390036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11569.286977] LNet: Removed LNI 192.168.204.139@tcp1 [11570.174377] Key type .llcrypt unregistered [11570.179672] Key type ._llcrypt unregistered [11588.320607] Key type ._llcrypt registered [11588.327582] Key type .llcrypt registered [11588.456577] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [11588.782433] Key type .llcrypt unregistered [11588.790607] Key type ._llcrypt unregistered [11596.574243] Key type ._llcrypt registered [11596.593184] Key type .llcrypt registered [11596.805523] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [11597.275779] Key type .llcrypt unregistered [11597.278692] Key type ._llcrypt unregistered [11613.141260] Key type ._llcrypt registered [11613.142569] Key type .llcrypt registered [11613.347212] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [11613.660604] Key type .llcrypt unregistered [11613.663703] Key type ._llcrypt unregistered [11637.733490] Key type ._llcrypt registered [11637.742954] Key type .llcrypt registered [11637.928908] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [11638.785344] Key type .llcrypt unregistered [11638.791953] Key type ._llcrypt unregistered [11657.272746] Key type ._llcrypt registered [11657.276116] Key type .llcrypt registered [11657.380057] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [11658.520470] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11658.735554] alg: No test for adler32 (adler32-zlib) [11659.863334] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [11660.278110] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [11661.983482] Key type lgssc registered [11663.433567] Lustre: Echo OBD driver; http://www.lustre.org/ [11687.955247] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing lnet_if_list [11689.717595] LNet: Added LNI 192.168.204.139@tcp1 [8/256/0/180] [11702.871492] loop0: detected capacity change from 0 to 204800000 [11703.114350] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11703.130525] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11703.168763] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11723.333913] loop0: detected capacity change from 0 to 204800000 [11723.658228] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11723.673855] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11723.694351] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11744.499962] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11744.535570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11745.905729] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11745.974862] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11746.402069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11752.540066] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [11762.193347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11762.298611] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11762.336970] Lustre: 395494:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11762.759326] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11768.425625] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [11775.716433] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11781.657807] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11787.937750] loop0: detected capacity change from 0 to 204800000 [11807.926544] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11808.464675] Lustre: sv26894-OST0000: new disk, initializing [11808.481799] Lustre: srv-sv26894-OST0000: No data found on store. Initialize space. [11808.505511] Lustre: 397573:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11808.673596] Lustre: sv26894-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11816.535212] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [11827.490655] loop1: detected capacity change from 0 to 204800000 [11846.316442] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11846.462944] Lustre: sv26894-OST0001: new disk, initializing [11846.467391] Lustre: srv-sv26894-OST0001: No data found on store. Initialize space. [11846.477326] Lustre: 398690:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11846.568662] Lustre: sv26894-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11852.903570] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [11867.480467] Lustre: 399681:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [11867.493954] Lustre: 399681:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [11867.501509] Lustre: 399681:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [11867.518632] Lustre: 399681:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [11867.530223] Lustre: 399681:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [11867.541807] Lustre: 399681:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [11869.464467] Lustre: server umount sv26894-OST0001 complete [11878.789142] Lustre: server umount sv26894-OST0000 complete [11890.657376] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11890.668069] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11890.685946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11893.730311] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11893.730937] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11895.332208] Lustre: server umount lustre-MDT0000 complete [11898.602868] LustreError: 396236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788165420 with bad export cookie 9365880809995769522 [11898.621131] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11898.621496] LustreError: 396236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11898.907154] Lustre: server umount lustre-MDT0001 complete [11909.382396] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [11912.141640] Key type lgssc unregistered [11912.383733] LNet: 401687:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11912.389725] LNetError: 401687:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11912.394180] LNet: Removed LNI 192.168.204.139@tcp [11913.446323] LNet: Removed LNI 192.168.204.139@tcp1 [11914.273648] Key type .llcrypt unregistered [11914.276900] Key type ._llcrypt unregistered [11935.169832] Key type ._llcrypt registered [11935.174402] Key type .llcrypt registered [11935.305231] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [11935.594281] Key type .llcrypt unregistered [11935.599180] Key type ._llcrypt unregistered [11951.369401] Key type ._llcrypt registered [11951.372382] Key type .llcrypt registered [11951.462069] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [11952.656033] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11952.694765] alg: No test for adler32 (adler32-zlib) [11953.760646] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [11953.991678] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [11955.647186] Key type lgssc registered [11956.877328] Lustre: Echo OBD driver; http://www.lustre.org/ [11978.572779] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing lnet_if_list [11979.667500] LNet: Added LNI 192.168.204.139@tcp1 [8/256/0/180] [11988.965985] loop0: detected capacity change from 0 to 204800000 [11989.111382] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11989.129619] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11989.141501] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12006.995872] loop0: detected capacity change from 0 to 204800000 [12007.172850] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12007.190549] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12007.215328] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12026.157414] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12026.172231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12027.530527] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12027.564421] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12027.755093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12032.220773] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [12041.240831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12041.355665] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12041.385246] Lustre: 405511:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12041.619712] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12046.569688] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [12054.103666] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12060.178757] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12064.693470] loop0: detected capacity change from 0 to 204800000 [12083.386399] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12083.740904] Lustre: sv15337-OST0000: new disk, initializing [12083.744713] Lustre: srv-sv15337-OST0000: No data found on store. Initialize space. [12083.755322] Lustre: 407588:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12083.820843] Lustre: sv15337-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12089.745568] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [12099.200618] loop1: detected capacity change from 0 to 204800000 [12118.185887] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12118.302783] Lustre: sv15337-OST0001: new disk, initializing [12118.308904] Lustre: srv-sv15337-OST0001: No data found on store. Initialize space. [12118.315739] Lustre: 408705:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12118.388536] Lustre: sv15337-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12124.576564] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [12139.883178] Lustre: 409695:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [12139.895429] Lustre: 409695:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [12139.901105] Lustre: 409695:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [12139.908809] Lustre: 409695:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [12139.918263] Lustre: 409695:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [12139.924380] Lustre: 409695:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [12142.340398] Lustre: server umount sv15337-OST0001 complete [12152.321962] Lustre: server umount sv15337-OST0000 complete [12164.593205] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12164.603279] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12164.623736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12168.673318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12168.675609] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12173.797845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12173.806580] Lustre: Skipped 2 previous similar messages [12178.399648] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12178.709272] Lustre: server umount lustre-MDT0000 complete [12182.574931] LustreError: 405503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788165704 with bad export cookie 5756863892142131948 [12182.594154] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12182.979837] Lustre: server umount lustre-MDT0001 complete [12195.445830] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [12198.819939] Key type lgssc unregistered [12199.125765] LNet: 411703:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12199.133677] LNetError: 411703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12199.140595] LNet: Removed LNI 192.168.204.139@tcp [12200.169642] LNet: Removed LNI 192.168.204.139@tcp1 [12201.028237] Key type .llcrypt unregistered [12201.032982] Key type ._llcrypt unregistered [12218.900734] Key type ._llcrypt registered [12218.903776] Key type .llcrypt registered [12219.012168] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [12219.289250] Key type .llcrypt unregistered [12219.293274] Key type ._llcrypt unregistered [12237.398160] Key type ._llcrypt registered [12237.400488] Key type .llcrypt registered [12237.514533] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [12239.053545] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12239.095059] alg: No test for adler32 (adler32-zlib) [12240.224269] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [12240.489117] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [12242.135185] Key type lgssc registered [12243.367938] Lustre: Echo OBD driver; http://www.lustre.org/ [12263.874327] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing lnet_if_list [12265.174650] LNet: Added LNI 192.168.204.139@tcp1 [8/256/0/180] [12274.646169] loop0: detected capacity change from 0 to 204800000 [12274.839523] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12274.853962] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12274.876430] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12293.014101] loop0: detected capacity change from 0 to 204800000 [12293.315381] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12293.337991] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12293.397674] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12312.777511] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12312.800162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12314.184509] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12314.220451] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12314.420224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12319.327672] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [12329.068103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12329.197161] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12329.231707] Lustre: 415529:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12329.601431] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12334.148900] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [12342.037155] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12347.252517] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12352.426037] loop0: detected capacity change from 0 to 204800000 [12371.312729] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12371.792419] Lustre: sv25641-OST0000: new disk, initializing [12371.802875] Lustre: srv-sv25641-OST0000: No data found on store. Initialize space. [12371.823921] Lustre: 417606:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12371.893759] Lustre: sv25641-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12378.515736] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [12390.233896] loop1: detected capacity change from 0 to 204800000 [12410.424940] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12410.683904] Lustre: sv25641-OST0001: new disk, initializing [12410.693791] Lustre: srv-sv25641-OST0001: No data found on store. Initialize space. [12410.704168] Lustre: 418724:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12410.794236] Lustre: sv25641-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12418.271844] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [12436.657655] Lustre: 419811:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [12436.669940] Lustre: 419811:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [12436.679234] Lustre: 419811:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [12436.684391] Lustre: 419811:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [12436.689765] Lustre: 419811:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [12436.695212] Lustre: 419811:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [12437.568177] Lustre: 419859:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [12437.579621] Lustre: 419859:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 255 previous similar messages [12437.611380] Lustre: 419859:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [12437.620605] Lustre: 419859:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 255 previous similar messages [12437.631784] Lustre: 419859:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [12437.641993] Lustre: 419859:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 255 previous similar messages [12437.655625] Lustre: 419859:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [12437.672327] Lustre: 419859:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 255 previous similar messages [12437.678842] Lustre: 419859:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [12437.687125] Lustre: 419859:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 255 previous similar messages [12437.696448] Lustre: 419859:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [12437.710914] Lustre: 419859:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 255 previous similar messages [12439.859528] Lustre: server umount sv25641-OST0001 complete [12450.845533] Lustre: server umount sv25641-OST0000 complete [12466.160155] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12466.175107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12466.203603] Lustre: Skipped 1 previous similar message [12466.233572] Lustre: Skipped 1 previous similar message [12471.270814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12471.281765] Lustre: Skipped 1 previous similar message [12471.992750] Lustre: server umount lustre-MDT0000 complete [12475.885165] LustreError: 415523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788165997 with bad export cookie 2496553408922832926 [12475.900579] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12475.906601] LustreError: 415523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12476.251035] Lustre: server umount lustre-MDT0001 complete [12490.043512] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [12492.964460] Key type lgssc unregistered [12493.318730] LNet: 421867:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12493.327598] LNetError: 421867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12494.367811] LNet: Removed LNI 192.168.204.139@tcp [12494.382012] LNetError: 421867:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12494.409728] LNet: Removed LNI 192.168.204.139@tcp1 [12495.323437] Key type .llcrypt unregistered [12495.328095] Key type ._llcrypt unregistered [12514.680695] Key type ._llcrypt registered [12514.683082] Key type .llcrypt registered [12514.856947] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [12515.186417] Key type .llcrypt unregistered [12515.188893] Key type ._llcrypt unregistered [12533.430504] Key type ._llcrypt registered [12533.433586] Key type .llcrypt registered [12533.547970] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [12535.432849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12535.486726] alg: No test for adler32 (adler32-zlib) [12536.682697] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [12537.162753] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [12538.943151] Key type lgssc registered [12540.458467] Lustre: Echo OBD driver; http://www.lustre.org/ [12566.763672] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing lnet_if_list [12568.114463] LNet: Added LNI 192.168.204.139@tcp1 [8/256/0/180] [12580.008257] loop0: detected capacity change from 0 to 204800000 [12580.270980] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12580.295107] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12580.319881] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12598.072910] loop0: detected capacity change from 0 to 204800000 [12598.371374] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12598.404248] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12598.429216] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12616.838922] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12616.874575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12618.080501] LNet: 1 peer NIs in recovery (showing 1): 192.168.204.139@tcp [12618.181857] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12618.204734] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12618.338276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12623.789508] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [12634.452055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12634.591035] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12634.635353] Lustre: 425691:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12635.008364] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12640.980810] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [12649.261616] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12654.805081] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12660.296191] loop0: detected capacity change from 0 to 204800000 [12679.533643] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12679.811824] Lustre: sv1746-OST0000: new disk, initializing [12679.815503] Lustre: srv-sv1746-OST0000: No data found on store. Initialize space. [12679.821951] Lustre: 427770:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12679.876333] Lustre: sv1746-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12685.840936] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [12701.725626] Lustre: 428711:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [12701.743226] Lustre: 428711:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [12701.748741] Lustre: 428711:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [12701.755806] Lustre: 428711:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [12701.765411] Lustre: 428711:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [12701.776701] Lustre: 428711:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [12708.051841] Lustre: server umount sv1746-OST0000 complete [12725.220199] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12725.258517] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12725.270550] Lustre: Skipped 1 previous similar message [12730.339325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12730.350245] Lustre: Skipped 1 previous similar message [12730.822168] Lustre: server umount lustre-MDT0000 complete [12734.785621] LustreError: 425685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788166256 with bad export cookie 6649427614974949912 [12734.788868] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12734.801089] LustreError: 425685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12735.175638] Lustre: server umount lustre-MDT0001 complete [12747.882337] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [12750.622619] Key type lgssc unregistered [12750.875592] LNet: 430418:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12750.885955] LNetError: 430418:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12750.891960] LNet: Removed LNI 192.168.204.139@tcp [12751.912175] LNet: Removed LNI 192.168.204.139@tcp1 [12752.943435] Key type .llcrypt unregistered [12752.946028] Key type ._llcrypt unregistered [12774.835581] Key type ._llcrypt registered [12774.838997] Key type .llcrypt registered [12774.991748] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [12775.304322] Key type .llcrypt unregistered [12775.306577] Key type ._llcrypt unregistered [12796.319650] Key type ._llcrypt registered [12796.322315] Key type .llcrypt registered [12796.507311] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [12797.902837] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12797.993489] alg: No test for adler32 (adler32-zlib) [12799.354491] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [12799.801707] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [12801.607264] Key type lgssc registered [12802.994959] Lustre: Echo OBD driver; http://www.lustre.org/ [12825.317677] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing lnet_if_list [12826.451478] LNet: Added LNI 192.168.204.139@tcp1 [8/256/0/180] [12828.877856] LNet: Added LNI 192.168.204.139@tcp2 [8/256/0/180] [12842.672972] loop0: detected capacity change from 0 to 204800000 [12843.033913] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12843.046742] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12843.067874] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12861.066786] loop0: detected capacity change from 0 to 204800000 [12861.213415] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12861.230548] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12861.246891] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12881.914684] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12881.951706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12883.358903] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12883.413946] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12883.638295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12889.266534] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [12898.303234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12898.432573] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12898.468496] Lustre: 434340:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12898.876778] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12904.159931] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [12912.285888] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12918.574849] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12923.607815] loop0: detected capacity change from 0 to 204800000 [12942.967457] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12943.399754] Lustre: sv7425-OST0000: new disk, initializing [12943.411732] Lustre: srv-sv7425-OST0000: No data found on store. Initialize space. [12943.426586] Lustre: 436417:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12943.523384] Lustre: sv7425-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12950.059129] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [12960.179425] loop1: detected capacity change from 0 to 204800000 [12978.943585] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12979.118995] Lustre: sv7425-OST0001: new disk, initializing [12979.129514] Lustre: srv-sv7425-OST0001: No data found on store. Initialize space. [12979.141439] Lustre: 437534:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12979.204960] Lustre: sv7425-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12985.497294] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [12993.399136] Lustre: DEBUG MARKER: SKIP: conf-sanity test_57p need a remote client with no Lustre server role for routing [12994.509342] Lustre: 438283:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [12994.516385] Lustre: 438283:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [12994.522508] Lustre: 438283:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [12994.526298] Lustre: 438283:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [12994.533173] Lustre: 438283:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [12994.540753] Lustre: 438283:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [12996.385870] Lustre: server umount sv7425-OST0001 complete [13006.065761] Lustre: server umount sv7425-OST0000 complete [13020.641936] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13020.660721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13020.669212] Lustre: Skipped 1 previous similar message [13025.273365] Lustre: server umount lustre-MDT0000 complete [13029.212469] LustreError: 437533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788166551 with bad export cookie 2380515536420709155 [13029.220934] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13029.240846] LustreError: 437533:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13029.710306] Lustre: server umount lustre-MDT0001 complete [13040.474975] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [13043.445927] Key type lgssc unregistered [13043.710106] LNet: 440289:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13043.728650] LNetError: 440289:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13043.742541] LNet: Removed LNI 192.168.204.139@tcp [13044.767795] LNet: Removed LNI 192.168.204.139@tcp1 [13044.778494] LNetError: 440289:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13044.789848] LNetError: 440289:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13044.811716] LNet: Removed LNI 192.168.204.139@tcp2 [13045.709337] Key type .llcrypt unregistered [13045.714268] Key type ._llcrypt unregistered [13064.537289] Key type ._llcrypt registered [13064.539680] Key type .llcrypt registered [13064.658415] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [13065.003415] Key type .llcrypt unregistered [13065.007396] Key type ._llcrypt unregistered [13082.402800] Key type ._llcrypt registered [13082.404532] Key type .llcrypt registered [13082.545637] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [13083.772098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13083.798350] alg: No test for adler32 (adler32-zlib) [13085.070249] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [13085.333734] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [13087.018097] Key type lgssc registered [13088.438864] Lustre: Echo OBD driver; http://www.lustre.org/ [13111.175430] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing lnet_if_list [13112.788346] LNet: Added LNI 192.168.204.139@tcp1 [8/256/0/180] [13123.944387] loop0: detected capacity change from 0 to 204800000 [13124.112160] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13124.132164] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13124.142968] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13145.238906] loop0: detected capacity change from 0 to 204800000 [13145.657888] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13145.680129] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13145.692445] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13165.286766] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13165.313098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13166.745709] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [13166.793051] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13166.956718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13172.015856] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [13181.123986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13181.323672] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13181.364190] Lustre: 444165:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13181.636581] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13186.708083] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [13195.122400] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13201.410953] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13206.860502] loop0: detected capacity change from 0 to 204800000 [13226.152379] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13226.698300] Lustre: sv15416-OST0000: new disk, initializing [13226.702601] Lustre: srv-sv15416-OST0000: No data found on store. Initialize space. [13226.708637] Lustre: 446244:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13226.770907] Lustre: sv15416-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13234.419228] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [13244.955374] loop1: detected capacity change from 0 to 204800000 [13264.844873] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13265.098565] Lustre: sv15416-OST0001: new disk, initializing [13265.114143] Lustre: srv-sv15416-OST0001: No data found on store. Initialize space. [13265.119815] Lustre: 447362:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13265.204677] Lustre: sv15416-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13271.612995] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [13284.411784] LustreError: 448254:0:(nodemap_handler.c:262:nodemap_disc_visibility_update_end_nm()) cfs_fail_timeout id 2404 sleeping for 15000ms [13299.423882] LustreError: 448254:0:(nodemap_handler.c:262:nodemap_disc_visibility_update_end_nm()) cfs_fail_timeout id 2404 awake [13308.583601] Lustre: 448547:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [13308.591913] Lustre: 448547:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [13308.600716] Lustre: 448547:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [13308.606140] Lustre: 448547:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [13308.613706] Lustre: 448547:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [13308.620306] Lustre: 448547:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [13310.768099] Lustre: server umount sv15416-OST0001 complete [13320.426326] Lustre: server umount sv15416-OST0000 complete [13335.520409] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13335.532946] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13335.574518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13337.581101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13337.585944] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13342.691572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13342.698374] Lustre: Skipped 2 previous similar messages [13347.812919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13347.837424] Lustre: Skipped 1 previous similar message [13349.344546] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13349.924215] Lustre: server umount lustre-MDT0000 complete [13355.298795] LustreError: 444158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788166877 with bad export cookie 17874666521041550455 [13355.301370] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13355.314590] LustreError: 444158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13355.710880] Lustre: server umount lustre-MDT0001 complete [13370.412184] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [13374.222683] Key type lgssc unregistered [13374.708586] LNet: 450568:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13374.714186] LNetError: 450568:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13374.728443] LNet: Removed LNI 192.168.204.139@tcp [13375.781864] LNet: Removed LNI 192.168.204.139@tcp1 [13377.038217] Key type .llcrypt unregistered [13377.050772] Key type ._llcrypt unregistered [13397.655930] Key type ._llcrypt registered [13397.657957] Key type .llcrypt registered [13397.747254] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [13398.015568] Key type .llcrypt unregistered [13398.021328] Key type ._llcrypt unregistered [13414.604837] Key type ._llcrypt registered [13414.607293] Key type .llcrypt registered [13414.688116] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [13415.742576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13415.886361] alg: No test for adler32 (adler32-zlib) [13416.965556] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [13417.232733] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [13419.031229] Key type lgssc registered [13420.030324] Lustre: Echo OBD driver; http://www.lustre.org/ [13442.452460] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing lnet_if_list [13443.645403] LNet: Added LNI 192.168.204.139@tcp1 [8/256/0/180] [13454.320103] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13454.349054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13455.689508] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [13455.719898] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13455.859235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13460.782286] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [13470.696328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13470.827818] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13470.875861] Lustre: 453836:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13471.264208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13476.788417] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [13485.051182] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13492.491556] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13500.299949] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing lnet_if_list [13500.707119] Lustre: DEBUG MARKER: rpc test_57r: @@@@@@ FAIL: Can't determine interface name for NID 192.168.204.139@tcp1 [13502.407730] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing check_logdir /tmp/test_logs/1788167021 [13512.164511] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13512.175952] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13512.190117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13517.794767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13517.802475] Lustre: Skipped 2 previous similar messages [13518.473497] Lustre: server umount lustre-MDT0000 complete [13522.686474] LustreError: 453827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788167044 with bad export cookie 5401532737560967719 [13522.688348] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13522.733738] LustreError: 453827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13523.082935] Lustre: server umount lustre-MDT0001 complete [13539.702601] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [13542.412298] Key type lgssc unregistered [13542.741593] LNet: 457182:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13542.762675] LNetError: 457182:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13542.770437] LNet: Removed LNI 192.168.204.139@tcp [13543.782650] LNet: Removed LNI 192.168.204.139@tcp1 [13544.666240] Key type .llcrypt unregistered [13544.676191] Key type ._llcrypt unregistered [13560.632473] Key type ._llcrypt registered [13560.634178] Key type .llcrypt registered [13560.732980] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [13561.822429] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13561.907558] alg: No test for adler32 (adler32-zlib) [13563.035621] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [13563.301893] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [13564.991199] Key type lgssc registered [13566.208339] Lustre: Echo OBD driver; http://www.lustre.org/ [13571.961989] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing lnet_if_list [13572.845867] LNet: Added LNI 192.168.204.139@tcp1 [8/256/0/180] [13579.381741] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13579.395796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13580.729833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13584.935706] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [13591.009701] LustreError: 458664:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13591.025394] LustreError: 458664:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13593.154986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13593.545529] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13598.104141] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [13604.975820] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13610.077096] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13616.738985] LNet: Removed LNI 192.168.204.139@tcp1 [13617.514663] Lustre: 460374:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [13617.520704] Lustre: 460374:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [13617.524950] Lustre: 460374:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [13617.535674] Lustre: 460374:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [13617.545742] Lustre: 460374:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [13617.550935] Lustre: 460374:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [13624.011069] Lustre: server umount lustre-MDT0000 complete [13624.290351] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13624.294533] LustreError: 458664:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13624.302895] Lustre: Skipped 1 previous similar message [13624.331848] LustreError: 458664:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13624.342900] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13627.715306] LustreError: 458647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788167149 with bad export cookie 2317533278908426049 [13627.724091] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13627.737256] LustreError: 458647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13627.770730] Lustre: 458657:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.204.139@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [13630.109746] Lustre: server umount lustre-MDT0001 complete [13640.277858] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [13642.837424] Key type lgssc unregistered [13643.147485] LNet: 461378:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13643.157506] LNetError: 461378:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13643.178332] LNet: Removed LNI 192.168.204.139@tcp [13644.051155] Key type .llcrypt unregistered [13644.053822] Key type ._llcrypt unregistered [13662.723065] Key type ._llcrypt registered [13662.727960] Key type .llcrypt registered [13662.834447] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [13663.546327] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13663.609300] alg: No test for adler32 (adler32-zlib) [13664.791778] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [13665.050781] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [13666.695218] Key type lgssc registered [13667.987456] Lustre: Echo OBD driver; http://www.lustre.org/ [13676.705291] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13676.746309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13678.317916] LNetError: 462020:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.139@tcp1 has no available nets [13678.383147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13683.263531] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [13688.814046] LustreError: 462710:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13688.839545] LustreError: 462710:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13692.565023] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13692.871401] LNetError: 462020:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.139@tcp1 has no available nets [13692.888051] LNetError: 462020:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 2 previous similar messages [13692.990784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13697.318641] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [13705.682652] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13711.982796] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13721.265396] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13721.619624] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [13721.834281] LNetError: 462020:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.139@tcp1 has no available nets [13721.843721] LNetError: 462020:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 2 previous similar messages [13721.885236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13729.314420] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [13731.845859] LNetError: 462020:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.139@tcp1 has no available nets [13731.862492] LNetError: 462020:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 2 previous similar messages [13738.414624] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13747.688346] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13747.695952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13747.716162] Lustre: Skipped 2 previous similar messages [13747.730700] Lustre: Skipped 2 previous similar messages [13751.632384] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.39@tcp (stopping) [13752.803305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13752.810239] Lustre: Skipped 2 previous similar messages [13753.369731] Lustre: server umount lustre-MDT0000 complete [13756.733943] LustreError: 462707:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13757.533139] LustreError: 462693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788167279 with bad export cookie 14664859187862179462 [13757.546897] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13757.548142] LustreError: 462693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13757.921406] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13757.923766] LustreError: 462706:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13757.948470] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13761.843265] LustreError: 465166:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13763.057914] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13763.063486] Lustre: Skipped 1 previous similar message [13764.015721] Lustre: server umount lustre-MDT0001 complete [13771.735964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [13780.856532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13782.114038] LustreError: 466304:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [13782.122145] LustreError: 466304:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@0000000061c83379 x1875029195273216/t0(0) o250->MGC192.168.204.139@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1788167304 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13782.148040] LustreError: 466304:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [13783.016388] LNetError: 462020:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.139@tcp1 has no available nets [13783.029199] LNetError: 462020:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 3 previous similar messages [13783.038486] LustreError: 464539:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13783.068695] LustreError: 464539:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13783.146540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13783.238720] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [13788.379349] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [13792.355238] LustreError: 466318:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13792.358249] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13792.393244] LustreError: 466318:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [13798.291033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13798.491458] LNetError: 462020:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.139@tcp1 has no available nets [13798.504390] LNetError: 462020:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 2 previous similar messages [13798.661299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13803.149893] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [13804.024592] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13811.743644] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13813.214550] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13818.065691] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13819.404475] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [13829.092531] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13829.109524] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13829.145229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13829.612642] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13830.990443] Lustre: server umount lustre-OST0000 complete [13839.846312] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13839.848916] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13839.862602] Lustre: Skipped 2 previous similar messages [13841.543954] Lustre: server umount lustre-MDT0000 complete [13844.963902] LustreError: 466322:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13844.984696] LustreError: 466322:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [13845.157773] LustreError: 464545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788167367 with bad export cookie 14664859187862187351 [13845.160433] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13845.167227] LustreError: 464545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13845.521648] Lustre: server umount lustre-MDT0001 complete [13858.416134] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [13861.873591] Key type lgssc unregistered [13862.276865] LNet: 468992:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13862.290298] LNetError: 468992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13862.311107] LNet: Removed LNI 192.168.204.139@tcp [13863.293926] Key type .llcrypt unregistered [13863.296456] Key type ._llcrypt unregistered [13888.643517] Key type ._llcrypt registered [13888.646166] Key type .llcrypt registered [13888.780460] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [13889.905294] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13889.959988] alg: No test for adler32 (adler32-zlib) [13891.073794] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [13891.433828] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [13893.127219] Key type lgssc registered [13894.555845] Lustre: Echo OBD driver; http://www.lustre.org/ [13901.811057] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13901.822181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13903.236187] LNetError: 469717:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.139@tcp1 has no available nets [13903.307220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13907.756640] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [13913.572991] LustreError: 470409:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13913.606175] LustreError: 470409:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13916.044303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13916.489895] LNetError: 469717:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.139@tcp1 has no available nets [13916.500705] LNetError: 469717:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 3 previous similar messages [13916.617591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13920.579615] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [13927.949841] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13933.293595] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13937.121179] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13937.138620] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13937.172140] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13939.168027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13939.169975] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13941.283406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13941.290642] Lustre: Skipped 2 previous similar messages [13942.581704] Lustre: server umount lustre-MDT0000 complete [13945.942108] LustreError: 470389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788167467 with bad export cookie 16989700353670343691 [13945.943849] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13945.952786] LustreError: 470389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13946.281720] Lustre: server umount lustre-MDT0001 complete [13960.575856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13960.746371] LNetError: 469717:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.139@tcp1 has no available nets [13960.758927] LNetError: 469717:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 3 previous similar messages [13960.869847] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [13960.899656] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13961.092468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13966.150266] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [13974.816792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13974.965306] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13974.999309] Lustre: 472997:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13975.130264] LNetError: 469717:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.139@tcp1 has no available nets [13975.140402] LNetError: 469717:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 1 previous similar message [13975.257422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13979.748517] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [13987.639891] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13994.267536] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14004.411464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14004.541545] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [14004.548053] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [14004.553536] LustreError: 474881:0:(tgt_mount.c:2688:server_fill_super()) Unable to start targets: -2 [14004.558538] LustreError: 474881:0:(tgt_mount.c:2206:server_put_super()) no obd lustre-OST0000 [14004.563101] LustreError: 474881:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [14004.617830] Lustre: server umount lustre-OST0000 complete [14004.620738] LustreError: 474881:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [14014.897419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14015.110723] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [14015.256637] LNetError: 469717:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.139@tcp1 has no available nets [14015.263894] LNetError: 469717:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 2 previous similar messages [14015.286992] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14021.099406] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [14024.713067] LNetError: 469717:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.139@tcp1 has no available nets [14024.728422] LNetError: 469717:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 1 previous similar message [14024.882758] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [14029.359498] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14037.073220] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14037.162558] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [14037.267911] Lustre: lustre-OST0001: new disk, initializing [14037.271822] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [14037.279207] Lustre: 476426:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14037.428862] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14044.248360] LNetError: 469717:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.204.139@tcp1 has no available nets [14044.269854] LNetError: 469717:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 5 previous similar messages [14044.349354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [14044.365179] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [14044.486536] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [14045.334940] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [14053.560468] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14060.015546] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14060.022594] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14060.027604] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [14060.029876] LustreError: Skipped 1 previous similar message [14060.054694] Lustre: Skipped 1 previous similar message [14062.523253] Lustre: server umount lustre-OST0001 complete [14065.126780] LustreError: 475348:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14065.151320] LustreError: 475348:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14070.253740] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14070.263772] LustreError: 475351:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14070.279057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14070.279068] Lustre: Skipped 1 previous similar message [14070.284586] Lustre: Skipped 1 previous similar message [14070.317943] LustreError: 475351:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14072.461879] Lustre: server umount lustre-OST0000 complete [14077.920503] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14077.929366] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14080.496545] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14080.500671] Lustre: Skipped 3 previous similar messages [14082.691331] Lustre: server umount lustre-MDT0000 complete [14085.623385] LustreError: 473005:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14085.645855] LustreError: 473005:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14086.821331] LustreError: 472989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788167608 with bad export cookie 16989700353670344244 [14086.833049] LustreError: 472989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14086.836594] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14087.282261] Lustre: server umount lustre-MDT0001 complete [14100.623877] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [14104.705833] Key type lgssc unregistered [14105.058329] LNet: 478385:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14105.067791] LNetError: 478385:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14105.121710] LNet: Removed LNI 192.168.204.139@tcp [14106.414963] Key type .llcrypt unregistered [14106.420123] Key type ._llcrypt unregistered [14189.123619] Key type ._llcrypt registered [14189.135183] Key type .llcrypt registered [14189.423186] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_hostid [14205.116453] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [14206.440923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14206.463273] alg: No test for adler32 (adler32-zlib) [14207.590361] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [14207.887503] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [14209.624208] Key type lgssc registered [14210.973114] Lustre: Echo OBD driver; http://www.lustre.org/ [14247.800516] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14247.868634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14249.141356] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14249.171218] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14249.247699] Lustre: lustre-MDT0000: new disk, initializing [14249.311161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14249.330423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14253.459950] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [14263.547119] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14263.662759] Lustre: 483323:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14263.739220] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14263.744590] Lustre: Skipped 1 previous similar message [14263.814783] Lustre: lustre-MDT0001: new disk, initializing [14263.928224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14263.992050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14264.008928] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14268.582928] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [14277.606756] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14283.726310] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14290.481352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14290.760730] Lustre: lustre-OST0000: new disk, initializing [14290.766647] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14290.774412] Lustre: 485293:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14290.881457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14296.650329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14296.660058] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14296.809633] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14297.824810] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [14307.324358] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14313.044252] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [14313.237924] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [14319.593626] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [14319.861273] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [14322.661622] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14322.665385] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14322.666552] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14322.710514] Lustre: Skipped 1 previous similar message [14327.780850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14327.802713] Lustre: Skipped 2 previous similar messages [14328.319946] Lustre: server umount lustre-OST0000 complete [14332.907810] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14332.915318] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14332.920805] Lustre: Skipped 1 previous similar message [14338.019506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14338.029708] Lustre: Skipped 2 previous similar messages [14343.137536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14343.142482] Lustre: Skipped 1 previous similar message [14347.231140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14347.498140] Lustre: server umount lustre-MDT0000 complete [14351.900427] LustreError: 483317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788167873 with bad export cookie 12676949914763356461 [14351.902095] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14351.912713] LustreError: 483317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14352.219436] Lustre: server umount lustre-MDT0001 complete [14361.738896] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 05:18:02 (1788167882) [14369.505861] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 05:18:10 (1788167890) [14379.782653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14380.263942] LustreError: 488051:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14380.289856] LustreError: 488051:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [14380.354768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14384.651566] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [14385.633534] LustreError: 488052:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14390.762954] LustreError: 488051:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14392.318157] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14392.643555] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14396.581300] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [14404.240752] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14409.798772] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14416.598642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14417.025052] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14423.200771] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [14430.109282] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14436.539752] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [14437.980166] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [14439.339333] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [14440.772681] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [14442.262111] Lustre: Failing over lustre-MDT0000 [14442.480465] Lustre: server umount lustre-MDT0000 complete [14444.003275] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14444.009950] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14444.023140] LustreError: 488781:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14447.081983] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14447.095911] Lustre: Skipped 1 previous similar message [14452.014309] LustreError: 490542:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [14452.033822] LustreError: 490542:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [14460.043735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14460.220143] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14460.460466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14460.515585] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [14462.278887] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [14464.998573] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [14465.521430] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [14465.558617] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [14465.620898] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [14475.829433] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14477.544856] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [14481.283421] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [14483.047602] Lustre: Failing over lustre-MDT0000 [14483.332727] Lustre: server umount lustre-MDT0000 complete [14485.987693] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14485.990460] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14486.000883] LustreError: 490542:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14486.011045] Lustre: Skipped 2 previous similar messages [14486.039715] LustreError: 490542:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [14501.341757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14501.343633] Lustre: 481546:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788168007/real 1788168007] req@000000009bb5b4ff x1875029764693504/t0(0) o400->MGC192.168.204.139@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788168023 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14501.376608] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14503.218927] LustreError: 488046:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [14503.237064] LustreError: 488046:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [14511.585447] LustreError: 481543:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000004c063ba8 x1875029764698496/t0(0) o250->MGC192.168.204.139@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14512.009058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14512.070203] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [14513.465983] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [14516.875927] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [14517.265063] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [14517.272628] Lustre: Skipped 2 previous similar messages [14517.307085] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [14517.361841] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [14526.561315] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14528.432528] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [14537.705027] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14537.716360] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14537.717521] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14537.739595] Lustre: Skipped 1 previous similar message [14542.683485] Lustre: server umount lustre-MDT0000 complete [14542.817849] LustreError: 488781:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14542.854908] LustreError: 488781:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [14547.044638] LustreError: 488033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788168068 with bad export cookie 12676949914763359177 [14547.057965] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14547.064831] LustreError: 488033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14547.781513] Lustre: server umount lustre-MDT0001 complete [14563.146928] Lustre: server umount lustre-OST0000 complete [14582.228730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14582.966301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14588.353138] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [14599.719788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14605.470482] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [14613.277529] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14619.815619] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14627.083515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14628.550293] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [14634.270934] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [14642.125330] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14653.932548] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [14653.942115] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14653.956596] Lustre: Skipped 1 previous similar message [14653.963171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14653.970422] Lustre: Skipped 2 previous similar messages [14667.232277] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14667.441667] Lustre: server umount lustre-OST0000 complete [14671.848904] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14671.864736] LustreError: Skipped 1 previous similar message [14677.662656] Lustre: server umount lustre-MDT0000 complete [14682.051634] LustreError: 494375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788168203 with bad export cookie 12676949914763359912 [14682.062562] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14682.063956] LustreError: 494375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14682.437963] Lustre: server umount lustre-MDT0001 complete [14695.094116] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [14698.015336] Key type lgssc unregistered [14698.348776] LNet: 497928:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14698.361553] LNetError: 497928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14698.381674] LNet: Removed LNI 192.168.204.139@tcp [14699.267545] Key type .llcrypt unregistered [14699.269356] Key type ._llcrypt unregistered [14731.598919] Key type ._llcrypt registered [14731.602804] Key type .llcrypt registered [14731.709439] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [14732.894462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14733.078443] alg: No test for adler32 (adler32-zlib) [14734.236937] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [14734.522930] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [14736.279571] Key type lgssc registered [14737.699528] Lustre: Echo OBD driver; http://www.lustre.org/ [14745.999417] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14746.038666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14747.485727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14752.470154] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [14757.857481] LustreError: 499449:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14757.886402] LustreError: 499449:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14761.527871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14761.950733] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14766.483360] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [14774.411599] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14780.351344] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14788.347392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14788.658762] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14792.761937] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [14795.201943] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [14803.801414] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14819.408123] Lustre: server umount lustre-MDT0000 complete [14822.387349] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14822.406640] Lustre: Skipped 2 previous similar messages [14822.410744] LustreError: 499444:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14822.463277] LustreError: 499444:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14823.581948] LustreError: 499429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788168345 with bad export cookie 8359714184265340655 [14823.608788] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14823.952680] Lustre: server umount lustre-MDT0001 complete [14838.707183] Lustre: server umount lustre-OST0000 complete [14855.140435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14855.756498] LustreError: 503182:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14855.847682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14859.964337] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [14861.293614] LustreError: 503183:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14865.393621] LustreError: 503182:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14870.019593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14870.341251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14876.544784] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [14886.576841] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14893.418135] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14901.960463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14902.484811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14909.913269] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [14918.411460] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14926.071985] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [14942.177495] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14942.182342] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14942.194911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14946.795370] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14946.800769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14946.837617] Lustre: Skipped 1 previous similar message [14948.154349] Lustre: server umount lustre-MDT0000 complete [14951.916489] LustreError: 503183:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14951.932566] LustreError: 503183:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14952.206780] LustreError: 503162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788168474 with bad export cookie 8359714184265341768 [14952.230069] LustreError: 503162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14952.242514] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14952.609497] Lustre: server umount lustre-MDT0001 complete [14966.616479] Lustre: server umount lustre-OST0000 complete [14985.814668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14986.605566] LustreError: 507116:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14986.695114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14991.799566] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [15001.393884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15007.011845] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [15015.134919] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15021.739928] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15028.782285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15029.396524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15029.407665] Lustre: Skipped 1 previous similar message [15031.480480] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [15036.524928] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [15044.922765] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15052.258377] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15052.264094] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15052.295706] Lustre: Skipped 1 previous similar message [15052.306489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15052.325294] Lustre: Skipped 1 previous similar message [15057.282356] Lustre: server umount lustre-OST0000 complete [15062.496678] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15062.534035] Lustre: Skipped 1 previous similar message [15062.540661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15062.555312] Lustre: Skipped 1 previous similar message [15067.616712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15067.633551] Lustre: Skipped 1 previous similar message [15068.743294] Lustre: server umount lustre-MDT0000 complete [15072.738252] LustreError: 509322:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15072.804890] LustreError: 509322:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [15074.127935] LustreError: 507097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788168596 with bad export cookie 8359714184265342797 [15074.147391] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15074.148671] LustreError: 507097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [15074.854225] Lustre: server umount lustre-MDT0001 complete [15091.694623] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [15095.058563] Key type lgssc unregistered [15095.470406] LNet: 510651:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15095.480538] LNetError: 510651:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15095.508864] LNet: Removed LNI 192.168.204.139@tcp [15096.443832] Key type .llcrypt unregistered [15096.452518] Key type ._llcrypt unregistered [15127.425534] Key type ._llcrypt registered [15127.426996] Key type .llcrypt registered [15127.579461] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [15128.924717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15129.074127] alg: No test for adler32 (adler32-zlib) [15130.227955] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [15130.545631] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [15132.327177] Key type lgssc registered [15133.398311] Lustre: Echo OBD driver; http://www.lustre.org/ [15143.080016] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15143.097450] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15143.116776] LustreError: 512093:0:(osd_handler.c:8554:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [15143.151979] LustreError: 512093:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [15143.158500] LustreError: 512093:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [15143.166181] LustreError: 512093:0:(tgt_mount.c:2656:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [15143.178892] LustreError: 512093:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [15152.724354] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [15152.728903] LustreError: 512491:0:(osd_handler.c:8554:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [15152.752424] LustreError: 512491:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [15152.757934] LustreError: 512491:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [15152.764824] LustreError: 512491:0:(tgt_mount.c:2656:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [15152.770685] LustreError: 512491:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [15175.348971] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [15178.589449] Key type lgssc unregistered [15178.979806] LNet: 513495:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15178.992645] LNetError: 513495:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15179.027967] LNet: Removed LNI 192.168.204.139@tcp [15180.247429] Key type .llcrypt unregistered [15180.252251] Key type ._llcrypt unregistered [15201.731380] Key type ._llcrypt registered [15201.734384] Key type .llcrypt registered [15201.844628] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_hostid [15217.272246] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [15218.494296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15218.539947] alg: No test for adler32 (adler32-zlib) [15219.811531] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [15220.068275] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [15221.783977] Key type lgssc registered [15223.242235] Lustre: Echo OBD driver; http://www.lustre.org/ [15260.644632] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15260.681957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15261.922308] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15261.957876] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15262.074279] Lustre: lustre-MDT0000: new disk, initializing [15262.177154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15262.211058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15266.825783] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [15276.826235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15276.976728] Lustre: 516655:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15277.060749] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15277.064292] Lustre: Skipped 1 previous similar message [15277.188216] Lustre: lustre-MDT0001: new disk, initializing [15277.308392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15277.345703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15277.366814] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15281.783984] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [15291.726602] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15298.059273] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15305.310562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15305.634728] Lustre: lustre-OST0000: new disk, initializing [15305.639819] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15305.654830] Lustre: 518624:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15305.758484] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15311.929384] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [15314.503170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15314.514791] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15314.641039] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15320.860415] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15325.487455] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15325.680947] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15330.205799] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15330.426844] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15335.392785] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15335.404109] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15335.435045] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15338.555983] Lustre: server umount lustre-OST0000 complete [15344.096957] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15344.103483] LustreError: Skipped 1 previous similar message [15344.107059] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15344.127144] Lustre: Skipped 1 previous similar message [15344.141867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15344.145424] Lustre: Skipped 1 previous similar message [15345.635632] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15345.640905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15345.652085] Lustre: Skipped 1 previous similar message [15350.758118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15350.770224] Lustre: Skipped 1 previous similar message [15355.874546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15355.882821] Lustre: Skipped 1 previous similar message [15356.383134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15356.530689] Lustre: server umount lustre-MDT0000 complete [15359.442659] LustreError: 516648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788168881 with bad export cookie 12360680963185793536 [15359.457137] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15359.461035] LustreError: 516648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15359.702580] Lustre: server umount lustre-MDT0001 complete [15367.269924] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [15368.747586] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 05:34:49 (1788168889) [15376.056995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15376.678605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15381.986195] LustreError: 521084:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15382.028515] LustreError: 521084:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [15382.518617] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [15387.106132] LustreError: 521083:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15392.229163] LustreError: 521084:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15393.591578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15394.177871] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15399.514986] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [15408.407532] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15414.162456] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15422.187260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15422.475275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15428.281395] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [15437.575777] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15446.052375] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15446.320167] Lustre: lustre-OST0001: new disk, initializing [15446.327338] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [15446.334956] Lustre: 523921:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15446.528863] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [15455.430611] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [15455.472646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [15455.503416] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [15455.641810] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [15465.999290] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15474.000042] Lustre: lustre-OST0001: Not available for connect from 192.168.204.39@tcp (stopping) [15476.194319] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15476.196092] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15476.210610] LustreError: Skipped 1 previous similar message [15476.230224] Lustre: Skipped 1 previous similar message [15477.755716] Lustre: server umount lustre-OST0001 complete [15479.095440] LustreError: 522908:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [15486.439455] LustreError: 522905:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15486.470585] LustreError: 522905:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [15496.686216] LustreError: 522907:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15496.695722] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15496.704857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15496.704868] Lustre: Skipped 2 previous similar messages [15496.704791] LustreError: 522907:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [15496.766544] Lustre: Skipped 1 previous similar message [15502.853441] Lustre: server umount lustre-OST0000 complete [15512.035874] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15512.039463] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15512.055393] Lustre: Skipped 1 previous similar message [15522.271895] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15522.614557] Lustre: server umount lustre-MDT0000 complete [15526.368877] LustreError: 521079:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15526.396240] LustreError: 521079:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [15527.313599] LustreError: 523920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788169049 with bad export cookie 12360680963185794418 [15527.339123] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15527.341333] LustreError: 523920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15533.838515] Lustre: server umount lustre-MDT0001 complete [15546.943265] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [15550.143478] Key type lgssc unregistered [15550.509697] LNet: 525913:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15550.526320] LNetError: 525913:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15550.546376] LNet: Removed LNI 192.168.204.139@tcp [15551.498989] Key type .llcrypt unregistered [15551.507696] Key type ._llcrypt unregistered [15606.737327] Key type ._llcrypt registered [15606.739983] Key type .llcrypt registered [15606.909643] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [15608.251845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15608.453945] alg: No test for adler32 (adler32-zlib) [15609.519458] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [15609.795545] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [15611.455508] Key type lgssc registered [15612.865763] Lustre: Echo OBD driver; http://www.lustre.org/ [15622.226836] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15622.260597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15623.613742] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [15623.651456] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15623.809188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15629.442865] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [15640.079326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15640.277641] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [15640.341468] Lustre: 528716:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15640.670260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15646.437332] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [15655.152049] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15661.545473] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15666.159540] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15666.168856] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15666.185337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15668.193263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15668.195310] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15670.616692] Lustre: server umount lustre-MDT0000 complete [15675.380418] LustreError: 528707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788169197 with bad export cookie 16926606724022093909 [15675.388711] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15675.404782] LustreError: 528707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15675.935626] Lustre: server umount lustre-MDT0001 complete [15688.422196] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 05:40:08 (1788169208) [15696.487191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15696.933841] LustreError: 531227:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15696.952542] LustreError: 531227:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [15697.001565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15702.160095] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [15702.498962] LustreError: 531228:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15707.625623] LustreError: 531227:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15711.451464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15711.824457] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15716.583393] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [15725.166670] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15731.544869] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15738.604447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15738.836860] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [15739.155290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15745.040166] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [15753.113390] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15761.445146] Lustre: Permanently deactivating lustre-OST0000 [15761.468657] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [15769.680656] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [15771.868280] LustreError: 534214:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [15771.890409] LustreError: 534214:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [15771.904608] LustreError: 534214:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [15775.793761] LustreError: 534263:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [15775.807692] LustreError: 534263:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [15780.343282] Lustre: server umount lustre-OST0000 complete [15783.732814] LustreError: 534512:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [15783.738954] LustreError: 534512:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [15788.512257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15788.520431] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15788.540604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15788.550127] Lustre: Skipped 1 previous similar message [15794.153392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15794.159711] Lustre: Skipped 3 previous similar messages [15799.268840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15799.276616] Lustre: Skipped 1 previous similar message [15800.287319] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15800.894913] Lustre: server umount lustre-MDT0000 complete [15804.390670] LustreError: 531227:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15804.408389] LustreError: 531227:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15806.012111] LustreError: 531208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788169327 with bad export cookie 16926606724022094490 [15806.023775] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15806.026808] LustreError: 531208:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15815.918742] Lustre: server umount lustre-MDT0001 complete [15826.734990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15832.416374] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [15835.890239] LustreError: 535673:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [15843.087216] Lustre: 536012:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.139@tcp [15845.531221] Lustre: server umount MGS complete [15860.890140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15861.505933] LustreError: 536738:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15861.527868] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [15861.534782] Lustre: Skipped 1 previous similar message [15861.580367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15867.626775] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [15871.971150] LustreError: 536737:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15872.000383] LustreError: 536737:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15878.925956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15879.239550] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [15879.326992] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15885.878561] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [15895.255369] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15902.497251] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15904.786301] Lustre: Permanently reactivating lustre-OST0000 [15904.796718] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [15904.801949] Lustre: Skipped 2 previous similar messages [15933.407215] LustreError: 538518:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [15942.695147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15943.148233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15951.088748] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [15960.418293] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15967.715951] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15967.721233] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15967.733122] Lustre: Skipped 1 previous similar message [15967.741928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15972.790719] Lustre: server umount lustre-OST0000 complete [15978.980448] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15979.004882] Lustre: Skipped 2 previous similar messages [15983.711698] Lustre: server umount lustre-MDT0000 complete [15984.097585] LustreError: 537466:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15984.119489] LustreError: 537466:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [15988.190978] LustreError: 536718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788169510 with bad export cookie 16926606724022095540 [15988.205418] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15988.206492] LustreError: 536718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15988.767281] Lustre: server umount lustre-MDT0001 complete [16001.703916] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [16005.099369] Key type lgssc unregistered [16005.422055] LNet: 541105:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16005.433419] LNetError: 541105:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16005.453863] LNet: Removed LNI 192.168.204.139@tcp [16006.434219] Key type .llcrypt unregistered [16006.438350] Key type ._llcrypt unregistered [16027.212416] Key type ._llcrypt registered [16027.231436] Key type .llcrypt registered [16027.406332] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_hostid [16046.353797] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [16047.712510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16047.990345] alg: No test for adler32 (adler32-zlib) [16049.259557] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [16049.650968] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [16051.367190] Key type lgssc registered [16053.098591] Lustre: Echo OBD driver; http://www.lustre.org/ [16096.293889] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16096.333323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16097.675614] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16097.722416] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16097.833404] Lustre: lustre-MDT0000: new disk, initializing [16097.988746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16098.008563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16104.223313] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [16116.723358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16116.853916] Lustre: 544266:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16116.913168] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16116.916713] Lustre: Skipped 1 previous similar message [16117.018215] Lustre: lustre-MDT0001: new disk, initializing [16117.123740] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16117.170265] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16117.184785] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16121.697175] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [16133.031699] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16140.296274] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16150.985510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16151.450416] Lustre: lustre-OST0000: new disk, initializing [16151.463830] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16151.479323] Lustre: 546235:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16151.660888] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16160.367372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16160.383314] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16160.629763] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16162.365389] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [16174.788855] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16181.069755] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16181.370344] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16186.426973] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16186.667735] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16191.460057] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16191.466236] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16191.471479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16191.482087] Lustre: Skipped 1 previous similar message [16191.495658] Lustre: Skipped 1 previous similar message [16196.580216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16196.585604] Lustre: Skipped 1 previous similar message [16201.701608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16201.704448] Lustre: Skipped 1 previous similar message [16203.231187] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16203.553520] Lustre: server umount lustre-OST0000 complete [16209.376260] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16209.386920] LustreError: Skipped 1 previous similar message [16209.395770] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16209.445382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16211.941748] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16215.145259] Lustre: server umount lustre-MDT0000 complete [16220.876616] LustreError: 544257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788169742 with bad export cookie 848277395321167115 [16220.899224] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16220.907942] LustreError: 544257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16221.489455] Lustre: server umount lustre-MDT0001 complete [16233.600844] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [16235.632463] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 05:49:15 (1788169755) [16244.024311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16244.590676] LustreError: 548699:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16244.610189] LustreError: 548699:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [16244.672158] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16249.586907] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [16249.828893] LustreError: 548700:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16254.951905] LustreError: 548699:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16259.011272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16259.331059] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16264.767652] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [16273.468170] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16278.972474] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16286.642691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16287.055274] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16294.912426] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [16305.139375] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16307.728682] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [16310.495852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [16319.970170] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16319.981093] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16319.984131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16320.004284] Lustre: Skipped 1 previous similar message [16320.025029] Lustre: Skipped 3 previous similar messages [16323.949339] Lustre: server umount lustre-OST0000 complete [16330.208600] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16330.223484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16330.228994] Lustre: Skipped 1 previous similar message [16334.186682] Lustre: server umount lustre-MDT0000 complete [16335.329188] LustreError: 548700:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16335.347725] LustreError: 548700:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16338.768612] LustreError: 548679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788169860 with bad export cookie 848277395321167997 [16338.774984] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16338.785349] LustreError: 548679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16339.302672] Lustre: server umount lustre-MDT0001 complete [16351.691223] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [16354.985879] Key type lgssc unregistered [16355.419666] LNet: 552427:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16355.430200] LNetError: 552427:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16355.451562] LNet: Removed LNI 192.168.204.139@tcp [16356.553826] Key type .llcrypt unregistered [16356.556585] Key type ._llcrypt unregistered [16390.976668] Key type ._llcrypt registered [16390.979315] Key type .llcrypt registered [16391.218897] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [16391.805653] Key type .llcrypt unregistered [16391.807381] Key type ._llcrypt unregistered [16417.157532] Key type ._llcrypt registered [16417.163318] Key type .llcrypt registered [16417.326931] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [16418.641591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16419.079312] alg: No test for adler32 (adler32-zlib) [16420.606387] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [16421.137040] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [16422.949318] Key type lgssc registered [16424.488694] Lustre: Echo OBD driver; http://www.lustre.org/ [16433.890060] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16433.917780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16435.511795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16441.312448] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [16445.938294] LustreError: 554643:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16445.978191] LustreError: 554643:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16450.667498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16451.055546] LustreError: 554644:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16451.247643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16456.677955] LustreError: 554640:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16456.708569] LustreError: 554640:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16459.119974] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [16460.847094] LustreError: 554643:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16460.874145] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [16460.874297] LustreError: 554643:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16465.890701] LustreError: 554640:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16465.901270] LustreError: 554640:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16468.110414] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16476.132651] LustreError: 554643:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16476.163859] LustreError: 554643:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [16476.432208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16476.784054] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16484.772370] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [16496.610986] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16496.621358] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16496.637934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16497.121951] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16497.147655] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16497.169362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16498.068392] Lustre: server umount lustre-OST0000 complete [16502.484859] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [16502.500941] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16502.546561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16502.563643] Lustre: Skipped 1 previous similar message [16502.964543] Lustre: server umount lustre-MDT0000 complete [16507.859980] LustreError: 554623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788170029 with bad export cookie 12535447475637000591 [16507.875814] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16507.884119] LustreError: 554623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16508.407601] Lustre: server umount lustre-MDT0001 complete [16520.713918] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [16524.383071] Key type lgssc unregistered [16524.773753] LNet: 557790:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16524.781574] LNetError: 557790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16524.804465] LNet: Removed LNI 192.168.204.139@tcp [16525.589267] Key type .llcrypt unregistered [16525.592204] Key type ._llcrypt unregistered [16553.720238] Key type ._llcrypt registered [16553.723410] Key type .llcrypt registered [16553.913395] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [16555.310672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16555.644098] alg: No test for adler32 (adler32-zlib) [16556.961868] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [16557.396574] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [16559.111307] Key type lgssc registered [16560.801459] Lustre: Echo OBD driver; http://www.lustre.org/ [16571.627725] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16571.683570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16587.426250] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [16592.545508] LustreError: 559187:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.139@tcp: failed processing log, type 4: rc = -110 [16619.231314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16627.395780] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [16635.907906] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16644.017704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16644.771621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16644.863787] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [16650.211960] LustreError: 559213:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16650.234831] LustreError: 559213:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16650.453335] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [16653.286558] LustreError: 559212:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16653.324596] LustreError: 559212:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16658.408104] LustreError: 560224:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16658.444190] LustreError: 560224:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16660.406406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16660.688831] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16665.757355] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [16674.718849] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16680.709392] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16690.527789] Lustre: server umount lustre-OST0000 complete [16695.807603] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [16695.817718] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16695.827802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16696.800942] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16696.801812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16696.818889] Lustre: Skipped 1 previous similar message [16701.922270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16701.934718] Lustre: Skipped 1 previous similar message [16705.075630] Lustre: server umount lustre-MDT0000 complete [16707.044580] LustreError: 561868:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16707.060459] LustreError: 561868:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16708.070826] Lustre: 558525:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788170213/real 1788170213] req@000000009df48c85 x1875032228789120/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788170229 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16708.071692] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16708.097906] Lustre: 558525:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [16708.666883] LustreError: 559193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788170230 with bad export cookie 3816663544455785349 [16708.675708] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16708.678861] LustreError: 559193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [16708.936462] Lustre: server umount lustre-MDT0001 complete [16719.121302] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [16722.067477] Key type lgssc unregistered [16722.396472] LNet: 562761:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16722.410574] LNetError: 562761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16723.430240] LNet: Removed LNI 192.168.204.139@tcp [16724.484817] Key type .llcrypt unregistered [16724.488222] Key type ._llcrypt unregistered [16751.446649] Key type ._llcrypt registered [16751.455255] Key type .llcrypt registered [16751.626976] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [16753.221603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16753.251555] alg: No test for adler32 (adler32-zlib) [16754.257468] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [16754.453922] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [16756.103222] Key type lgssc registered [16757.196892] Lustre: Echo OBD driver; http://www.lustre.org/ [16767.348645] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16767.399600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16768.993915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16774.376730] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [16778.212153] LustreError: 564180:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16778.239082] LustreError: 564180:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16783.331524] LustreError: 564181:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16783.349933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16783.878108] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16789.166380] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [16796.518158] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16801.963502] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16808.908781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16809.285260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16813.433550] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [16815.855685] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [16823.982890] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16829.922079] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16829.933676] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16829.945745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16831.750225] Lustre: server umount lustre-MDT0000 complete [16831.852659] LustreError: 564176:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [16834.031493] LustreError: 564181:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16834.042130] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16834.062750] LustreError: 564181:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16834.093153] Lustre: Skipped 1 previous similar message [16834.979987] LustreError: 564161:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.39@tcp arrived at 1788170356 with bad export cookie 4844637698982000122 [16837.421972] LustreError: 564162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788170359 with bad export cookie 4844637698981999233 [16837.432499] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16837.448484] LustreError: 564162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16839.138278] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16839.138937] LustreError: 564176:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16839.151939] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16839.159710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16839.223322] LustreError: 564176:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16842.145269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16843.915037] Lustre: server umount lustre-OST0000 complete [16847.284376] LustreError: 564180:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16847.315706] LustreError: 564180:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [16852.588118] Lustre: server umount lustre-MDT0001 complete [16865.143877] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [16868.464984] Key type lgssc unregistered [16868.902094] LNet: 567866:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16868.912181] LNetError: 567866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16868.942154] LNet: Removed LNI 192.168.204.139@tcp [16870.088867] Key type .llcrypt unregistered [16870.093650] Key type ._llcrypt unregistered [16901.642337] Key type ._llcrypt registered [16901.645356] Key type .llcrypt registered [16901.881872] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [16903.746473] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16904.096331] alg: No test for adler32 (adler32-zlib) [16905.438099] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [16905.810033] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [16907.591930] Key type lgssc registered [16908.930260] Lustre: Echo OBD driver; http://www.lustre.org/ [16917.018961] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16917.051429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16918.661180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16923.944738] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [16928.239296] LustreError: 569285:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16928.279711] LustreError: 569285:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16933.346099] LustreError: 569286:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16934.674108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16935.687202] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16940.984948] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [16952.325678] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16959.624354] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16967.858918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16968.517603] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16975.933241] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [16983.794953] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16990.009832] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [16999.751968] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.39@tcp (stopping) [17000.418258] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17000.422714] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [17000.430550] Lustre: Skipped 1 previous similar message [17000.440558] Lustre: Skipped 1 previous similar message [17004.161671] Lustre: server umount lustre-MDT0001 complete [17004.846951] LustreError: 571138:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [17010.662064] LustreError: 569285:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17010.679459] LustreError: 569285:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [17015.776141] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [17015.779267] LustreError: 569286:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17015.789093] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17015.829845] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17019.385169] Lustre: server umount lustre-OST0000 complete [17025.304165] Lustre: server umount lustre-MDT0000 complete [17042.912357] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [17045.997466] Key type lgssc unregistered [17046.362976] LNet: 573008:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17046.370653] LNetError: 573008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17047.408958] LNet: Removed LNI 192.168.204.139@tcp [17048.319236] Key type .llcrypt unregistered [17048.321362] Key type ._llcrypt unregistered [17074.133223] Key type ._llcrypt registered [17074.139396] Key type .llcrypt registered [17074.259416] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_hostid [17089.636529] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [17090.782551] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [17090.824151] alg: No test for adler32 (adler32-zlib) [17091.919854] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [17092.217084] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [17093.887198] Key type lgssc registered [17095.356420] Lustre: Echo OBD driver; http://www.lustre.org/ [17129.484343] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [17146.261361] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [17146.284400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17147.594254] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [17147.671848] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [17147.818788] Lustre: lustre-MDT0000: new disk, initializing [17147.928363] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17147.994023] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [17153.452192] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [17171.120833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17171.433107] Lustre: lustre-OST0000: new disk, initializing [17171.440763] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [17171.446744] Lustre: Skipped 1 previous similar message [17171.452277] Lustre: 578045:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [17171.541395] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17178.213913] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [17178.687948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [17178.695446] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [17178.725414] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [17188.185464] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17202.722927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17202.899804] Lustre: 577028:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [17202.975982] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [17203.031768] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.39@tcp (not set up) [17203.101273] Lustre: lustre-MDT0001: new disk, initializing [17203.272153] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17203.316365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [17203.329458] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [17208.367478] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [17217.979120] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [17218.278756] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [17223.263691] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [17223.613620] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [17233.763634] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.39@tcp (stopping) [17233.922335] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [17233.928868] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17233.948815] Lustre: Skipped 1 previous similar message [17236.177680] Lustre: server umount lustre-MDT0001 complete [17244.132170] LustreError: 577033:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17244.165924] LustreError: 577033:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [17249.251431] LustreError: 577034:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17249.273057] LustreError: 577034:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [17254.375327] LustreError: 577039:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17255.394282] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [17255.401529] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17255.418433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17255.432704] Lustre: Skipped 2 previous similar messages [17259.494501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17259.498386] LustreError: 577038:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17260.665518] Lustre: server umount lustre-OST0000 complete [17270.921694] Lustre: server umount lustre-MDT0000 complete [17284.342659] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [17287.241811] Key type lgssc unregistered [17287.568204] LNet: 581792:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17287.578599] LNetError: 581792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17287.602796] LNet: Removed LNI 192.168.204.139@tcp [17288.528263] Key type .llcrypt unregistered [17288.531673] Key type ._llcrypt unregistered [17327.256294] Key type ._llcrypt registered [17327.260489] Key type .llcrypt registered [17327.391676] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [17327.686652] Key type .llcrypt unregistered [17327.689980] Key type ._llcrypt unregistered [17342.157542] Key type ._llcrypt registered [17342.161109] Key type .llcrypt registered [17342.282912] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [17343.063557] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [17343.137060] alg: No test for adler32 (adler32-zlib) [17344.174114] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [17344.392719] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [17346.039170] Key type lgssc registered [17346.933301] Lustre: Echo OBD driver; http://www.lustre.org/ [17369.844783] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [17369.916578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17371.246927] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [17371.302715] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [17371.405206] Lustre: lustre-MDT0000: new disk, initializing [17371.510720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17371.543849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [17375.653299] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [17386.202148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17386.336521] Lustre: 585168:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [17386.385901] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [17386.390327] Lustre: Skipped 1 previous similar message [17386.468819] Lustre: lustre-MDT0001: new disk, initializing [17386.550214] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17386.599792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [17386.610844] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [17391.195637] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [17400.599168] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [17406.940755] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [17413.614799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17413.939391] Lustre: lustre-OST0000: new disk, initializing [17413.944889] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [17413.955857] Lustre: 587138:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [17414.037746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17420.026876] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [17423.422584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [17423.442336] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [17423.511492] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [17428.065759] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17437.665397] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [17437.668836] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17437.678410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17439.204648] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17439.205964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17439.232682] Lustre: Skipped 2 previous similar messages [17441.612402] Lustre: server umount lustre-MDT0000 complete [17445.748780] LustreError: 585162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788170967 with bad export cookie 4220383419377161478 [17445.753481] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17445.759133] LustreError: 585162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [17446.125528] Lustre: server umount lustre-MDT0001 complete [17460.359471] Lustre: server umount lustre-OST0000 complete [17471.667972] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 06:09:52 (1788170992) [17479.506506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17479.904458] LustreError: 589167:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17479.925231] LustreError: 589167:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [17479.976981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17484.492101] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [17485.284236] LustreError: 589168:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17489.377234] LustreError: 589167:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17492.609268] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17493.084830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17497.386585] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [17505.716500] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [17511.759980] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [17531.561136] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17531.811924] Lustre: Found index 0 for lustre-OST0000, updating log [17532.069262] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17538.685923] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [17546.445377] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17550.512946] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [17557.984514] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [17557.999244] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17558.024073] Lustre: Skipped 1 previous similar message [17558.032753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17562.744672] Lustre: server umount lustre-OST0000 complete [17577.805135] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17577.811961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17577.819838] Lustre: Skipped 2 previous similar messages [17577.829771] Lustre: Skipped 4 previous similar messages [17583.072757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17583.078104] Lustre: Skipped 1 previous similar message [17591.266104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17591.273305] Lustre: Skipped 5 previous similar messages [17592.287126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [17592.706667] Lustre: server umount lustre-MDT0000 complete [17594.272124] LNet: 1 local NIs in recovery (showing 1): 192.168.204.139@tcp [17596.068710] LustreError: 589147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788171117 with bad export cookie 4220383419377162871 [17596.069549] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17596.089808] LustreError: 589147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [17596.401915] Lustre: server umount lustre-MDT0001 complete [17605.330807] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 06:12:05 (1788171125) [18221.324607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18221.767234] LustreError: 595580:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18221.828726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [18226.453966] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [18227.169198] LustreError: 595581:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18231.268383] LustreError: 595580:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18234.383621] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18234.709545] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [18238.622426] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [18245.163917] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [18250.221755] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [18269.171095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [18269.380244] Lustre: Found index 0 for lustre-OST0000, updating log [18269.673810] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [18272.162042] LNet: 1 local NIs in recovery (showing 1): 192.168.204.139@tcp [18274.832393] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [18275.970362] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [18283.283687] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [18286.118314] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18286.138873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [18299.359145] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [18299.551589] Lustre: server umount lustre-OST0000 complete [18304.034631] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18304.058018] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [18304.064053] Lustre: Skipped 1 previous similar message [18308.200293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [18308.223772] Lustre: Skipped 7 previous similar messages [18316.387797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [18316.393304] Lustre: Skipped 15 previous similar messages [18317.279157] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [18317.414669] LustreError: 595580:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18317.432897] LustreError: 595580:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [18317.501777] Lustre: server umount lustre-MDT0000 complete [18321.084067] LustreError: 597435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788171842 with bad export cookie 4220383419377163956 [18321.090201] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [18321.099281] LustreError: 597435:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [18321.387837] Lustre: server umount lustre-MDT0001 complete [18332.575318] LNet: 1 local NIs in recovery (showing 1): 192.168.204.139@tcp [18350.463932] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_hostid [18359.433542] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [18394.687188] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 06:25:14 (1788171914) [18411.844362] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [18414.820320] Key type lgssc unregistered [18415.199392] LNet: 602197:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [18415.219966] LNetError: 602197:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [18532.004864] LNet: Removed LNI 192.168.204.139@tcp [18532.810449] Key type .llcrypt unregistered [18532.818520] Key type ._llcrypt unregistered [18548.605637] Key type ._llcrypt registered [18548.608844] Key type .llcrypt registered [18548.742721] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [18549.942595] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [18549.994087] alg: No test for adler32 (adler32-zlib) [18551.106196] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [18551.389858] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [18553.104135] Key type lgssc registered [18554.132737] Lustre: Echo OBD driver; http://www.lustre.org/ [18557.886397] LNet: Added LNI 192.168.204.139@tcp100 [8/256/0/180] [18558.727712] LNet: Added LNI 192.168.204.139@tcp101 [8/256/0/180] [18559.436806] LNet: Added LNI 192.168.204.139@tcp102 [8/256/0/180] [18560.156977] LNet: Added LNI 192.168.204.139@tcp103 [8/256/0/180] [18560.941821] LNet: Added LNI 192.168.204.139@tcp104 [8/256/0/180] [18561.790656] LNet: Added LNI 192.168.204.139@tcp105 [8/256/0/180] [18562.569595] LNet: Added LNI 192.168.204.139@tcp106 [8/256/0/180] [18563.281690] LNet: Added LNI 192.168.204.139@tcp107 [8/256/0/180] [18563.959868] LNet: Added LNI 192.168.204.139@tcp108 [8/256/0/180] [18564.662725] LNet: Added LNI 192.168.204.139@tcp109 [8/256/0/180] [18565.400732] LNet: Added LNI 192.168.204.139@tcp110 [8/256/0/180] [18566.204945] LNet: Added LNI 192.168.204.139@tcp111 [8/256/0/180] [18567.142949] LNet: Added LNI 192.168.204.139@tcp112 [8/256/0/180] [18568.124467] LNet: Added LNI 192.168.204.139@tcp113 [8/256/0/180] [18569.152085] LNet: Added LNI 192.168.204.139@tcp114 [8/256/0/180] [18570.057375] LNet: Added LNI 192.168.204.139@tcp115 [8/256/0/180] [18571.017402] LNet: Added LNI 192.168.204.139@tcp116 [8/256/0/180] [18572.090861] LNet: Added LNI 192.168.204.139@tcp117 [8/256/0/180] [18573.029038] LNet: Added LNI 192.168.204.139@tcp118 [8/256/0/180] [18573.821577] LNet: Added LNI 192.168.204.139@tcp119 [8/256/0/180] [18574.800833] LNet: Added LNI 192.168.204.139@tcp120 [8/256/0/180] [18575.986205] LNet: Added LNI 192.168.204.139@tcp121 [8/256/0/180] [18577.026661] LNet: Added LNI 192.168.204.139@tcp122 [8/256/0/180] [18578.084851] LNet: Added LNI 192.168.204.139@tcp123 [8/256/0/180] [18579.166576] LNet: Added LNI 192.168.204.139@tcp124 [8/256/0/180] [18580.073734] LNet: Added LNI 192.168.204.139@tcp125 [8/256/0/180] [18580.936331] LNet: Added LNI 192.168.204.139@tcp126 [8/256/0/180] [18581.710524] LNet: Added LNI 192.168.204.139@tcp127 [8/256/0/180] [18582.683939] LNet: Added LNI 192.168.204.139@tcp128 [8/256/0/180] [18583.395383] LNet: Added LNI 192.168.204.139@tcp129 [8/256/0/180] [18584.183141] LNet: Added LNI 192.168.204.139@tcp130 [8/256/0/180] [18585.391193] LNet: Added LNI 192.168.204.139@tcp131 [8/256/0/180] [18586.662103] LNet: Added LNI 192.168.204.139@tcp132 [8/256/0/180] [18587.763849] LNet: Added LNI 192.168.204.139@tcp133 [8/256/0/180] [18589.025804] LNet: Added LNI 192.168.204.139@tcp134 [8/256/0/180] [18590.448843] LNet: Added LNI 192.168.204.139@tcp135 [8/256/0/180] [18601.178468] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [18601.221219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18602.366632] Lustre: 605232:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.139@tcp: can't add NID 192.168.204.139@tcp130: rc = -75 [18602.454970] LNetError: 602691:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.139 [18602.482403] LNetError: Protocol error connecting to 192.168.204.139@tcp116 at host 192.168.204.139:988: is it running a compatible version of Lustre? [18602.496186] LNetError: Skipped 9 previous similar messages [18602.561460] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [18602.591193] LNet: 6 local NIs in recovery (showing 6): 192.168.204.139@tcp124, 192.168.204.139@tcp125, 192.168.204.139@tcp127, 192.168.204.139@tcp128, 192.168.204.139@tcp126, 192.168.204.139@tcp129 [18602.598815] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [18602.703518] Lustre: lustre-MDT0000: new disk, initializing [18602.822133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [18602.852720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [18603.558946] LNetError: 605263:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.139 [18603.573615] LNetError: 605263:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 10 previous similar messages [18603.603800] LNetError: Protocol error connecting to 192.168.204.139@tcp110 at host 192.168.204.139:988: is it running a compatible version of Lustre? [18605.669501] LNetError: 605266:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.139 [18605.684734] LNetError: 605266:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 2 previous similar messages [18605.690393] LNetError: Protocol error connecting to 192.168.204.139@tcp124 at host 192.168.204.139:988: is it running a compatible version of Lustre? [18605.697270] LNetError: Skipped 2 previous similar messages [18607.699441] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [18609.760572] LNetError: 602690:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.139 [18609.780347] LNetError: Protocol error connecting to 192.168.204.139@tcp124 at host 192.168.204.139:988: is it running a compatible version of Lustre? [18617.956283] LNetError: 605257:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.139 [18617.977978] LNetError: Protocol error connecting to 192.168.204.139@tcp124 at host 192.168.204.139:988: is it running a compatible version of Lustre? [18620.080778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18620.233871] Lustre: 605249:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [18620.306508] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [18620.311561] Lustre: Skipped 1 previous similar message [18620.406090] Lustre: lustre-MDT0001: new disk, initializing [18620.477570] Lustre: 606052:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.139@tcp: can't add NID 192.168.204.139@tcp130: rc = -75 [18620.486188] Lustre: 606052:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [18620.612994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [18620.673976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [18620.686847] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [18625.560202] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [18627.169559] Lustre: 606384:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.139@tcp: can't add NID 192.168.204.139@tcp130: rc = -75 [18627.182307] Lustre: 606384:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [18632.289741] LNetError: 602692:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.139 [18632.314512] LNetError: Protocol error connecting to 192.168.204.139@tcp110 at host 192.168.204.139:988: is it running a compatible version of Lustre? [18636.277404] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [18641.684819] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [18647.748466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [18648.015382] Lustre: lustre-OST0000: new disk, initializing [18648.018656] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [18648.024663] Lustre: 607224:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [18648.119962] Lustre: 607229:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.139@tcp: can't add NID 192.168.204.139@tcp130: rc = -75 [18648.125768] Lustre: 607229:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [18648.171972] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [18648.675444] LNetError: 602690:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.139 [18648.694612] LNetError: 602690:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 6 previous similar messages [18648.712251] LNetError: Protocol error connecting to 192.168.204.139@tcp121 at host 192.168.204.139:988: is it running a compatible version of Lustre? [18648.728991] LNetError: Skipped 6 previous similar messages [18653.955560] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [18654.256556] Lustre: 607592:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.139@tcp: can't add NID 192.168.204.139@tcp130: rc = -75 [18654.275802] Lustre: 607592:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [18654.390876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [18654.403160] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [18654.535551] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [18663.013042] LNet: 1 local NIs in recovery (showing 1): 192.168.204.139@tcp124 [18664.662931] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [18675.168259] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [18675.169889] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18675.190892] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [18675.198024] Lustre: Skipped 1 previous similar message [18675.208436] Lustre: Skipped 1 previous similar message [18678.267614] Lustre: server umount lustre-OST0000 complete [18682.337553] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [18682.349964] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18682.366216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [18683.490371] LNetError: 605252:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.139 [18683.503172] LNetError: 605252:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 11 previous similar messages [18683.514330] LNetError: Protocol error connecting to 192.168.204.139@tcp110 at host 192.168.204.139:988: is it running a compatible version of Lustre? [18683.530978] LNetError: Skipped 11 previous similar messages [18685.413074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [18685.415445] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18690.530098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [18690.541977] Lustre: Skipped 2 previous similar messages [18695.649130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [18696.159507] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [18696.560283] Lustre: server umount lustre-MDT0000 complete [18700.362721] LustreError: 605242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788172222 with bad export cookie 1573198424797153555 [18700.374366] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [18700.734994] Lustre: server umount lustre-MDT0001 complete [18711.907927] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [18714.692113] Key type lgssc unregistered [18715.147082] LNet: 609139:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [18715.154633] LNetError: 609139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [18715.162298] LNet: Removed LNI 192.168.204.139@tcp [18716.192602] LNet: Removed LNI 192.168.204.139@tcp100 [18716.204309] LNetError: 609139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [18716.216049] LNetError: 609139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [18716.228170] LNet: Removed LNI 192.168.204.139@tcp101 [18717.279409] LNet: Removed LNI 192.168.204.139@tcp102 [18717.286082] LNetError: 609139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [18717.297988] LNetError: 609139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [18717.317441] LNet: Removed LNI 192.168.204.139@tcp103 [18718.367215] LNet: Removed LNI 192.168.204.139@tcp104 [18719.393652] LNet: Removed LNI 192.168.204.139@tcp105 [18719.402393] LNetError: 609139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [18719.409129] LNetError: 609139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [18719.417303] LNet: Removed LNI 192.168.204.139@tcp106 [18720.479185] LNet: Removed LNI 192.168.204.139@tcp107 [18721.503222] LNet: Removed LNI 192.168.204.139@tcp108 [18722.530087] LNet: Removed LNI 192.168.204.139@tcp109 [18723.552414] LNet: Removed LNI 192.168.204.139@tcp110 [18723.555064] LNetError: 609139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [18723.560380] LNetError: 609139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [18723.565305] LNet: Removed LNI 192.168.204.139@tcp111 [18724.575147] LNet: Removed LNI 192.168.204.139@tcp112 [18725.599244] LNet: Removed LNI 192.168.204.139@tcp113 [18726.623389] LNet: Removed LNI 192.168.204.139@tcp114 [18727.647300] LNet: Removed LNI 192.168.204.139@tcp115 [18728.671331] LNet: Removed LNI 192.168.204.139@tcp116 [18729.696610] LNet: Removed LNI 192.168.204.139@tcp117 [18730.719223] LNet: Removed LNI 192.168.204.139@tcp118 [18731.743664] LNet: Removed LNI 192.168.204.139@tcp119 [18731.747339] LNetError: 609139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [18731.754433] LNetError: 609139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [18731.758799] LNet: Removed LNI 192.168.204.139@tcp120 [18732.767264] LNet: Removed LNI 192.168.204.139@tcp121 [18733.792772] LNet: Removed LNI 192.168.204.139@tcp122 [18734.815156] LNet: Removed LNI 192.168.204.139@tcp123 [18734.822213] LNet: Removed LNI 192.168.204.139@tcp124 [18735.839238] LNet: Removed LNI 192.168.204.139@tcp125 [18736.863601] LNet: Removed LNI 192.168.204.139@tcp126 [18737.887772] LNet: Removed LNI 192.168.204.139@tcp127 [18738.911250] LNet: Removed LNI 192.168.204.139@tcp128 [18739.935229] LNet: Removed LNI 192.168.204.139@tcp129 [18739.938099] LNet: Removed LNI 192.168.204.139@tcp130 [18739.941804] LNet: Removed LNI 192.168.204.139@tcp131 [18739.944715] LNet: Removed LNI 192.168.204.139@tcp132 [18739.947961] LNet: Removed LNI 192.168.204.139@tcp133 [18739.951121] LNet: Removed LNI 192.168.204.139@tcp134 [18739.960932] LNet: Removed LNI 192.168.204.139@tcp135 [18740.796272] Key type .llcrypt unregistered [18740.798531] Key type ._llcrypt unregistered [18764.964547] Key type ._llcrypt registered [18764.967364] Key type .llcrypt registered [18765.026812] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_hostid [18783.446856] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [18784.801048] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [18784.849166] alg: No test for adler32 (adler32-zlib) [18786.089032] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [18786.560218] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [18788.455174] Key type lgssc registered [18789.701717] Lustre: Echo OBD driver; http://www.lustre.org/ [18826.212406] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 06:32:26 (1788172346) [18846.573190] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [18849.783377] Key type lgssc unregistered [18850.100957] LNet: 613230:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [18850.113217] LNetError: 613230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [18850.139070] LNet: Removed LNI 192.168.204.139@tcp [18850.984532] Key type .llcrypt unregistered [18850.987691] Key type ._llcrypt unregistered [18867.261526] Key type ._llcrypt registered [18867.263764] Key type .llcrypt registered [18867.413924] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [18868.907197] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [18869.006975] alg: No test for adler32 (adler32-zlib) [18870.266976] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [18870.557414] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [18872.232308] Key type lgssc registered [18873.391537] Lustre: Echo OBD driver; http://www.lustre.org/ [18881.597569] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [18881.625687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18882.793374] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [18882.835612] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [18882.894370] Lustre: lustre-MDT0000: new disk, initializing [18882.991206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [18883.008175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [18886.781814] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [18896.995131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18897.208933] Lustre: 614443:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [18897.310639] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [18897.316848] Lustre: Skipped 1 previous similar message [18897.401567] Lustre: lustre-MDT0001: new disk, initializing [18897.477096] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [18897.508247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [18897.532032] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [18902.139531] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [18912.876667] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [18920.078023] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [18926.876610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [18927.192125] Lustre: lustre-OST0000: new disk, initializing [18927.198052] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [18927.210204] Lustre: 616414:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [18927.289332] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [18933.485868] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [18936.365068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [18936.384895] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [18936.481082] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [18942.357211] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [18946.342987] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing lnet_if_list [18951.485994] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [18951.505838] LNet: Added LNI 192.168.204.139@tcp10 [8/256/0/180] [18957.727475] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [18957.769561] LNet: Added LNI 192.168.204.139@tcp11 [8/256/0/180] [18963.591787] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [18963.630338] LNet: Added LNI 192.168.204.139@tcp12 [8/256/0/180] [18969.447787] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [18969.472895] LNet: Added LNI 192.168.204.139@tcp13 [8/256/0/180] [18974.736175] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [18974.805897] LNet: Added LNI 192.168.204.139@tcp14 [8/256/0/180] [18980.794878] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [18980.837788] LNet: Added LNI 192.168.204.139@tcp15 [8/256/0/180] [18985.622862] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [18985.670652] LNet: Added LNI 192.168.204.139@tcp16 [8/256/0/180] [18990.500839] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [18990.531628] LNet: Added LNI 192.168.204.139@tcp17 [8/256/0/180] [18995.428175] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [18995.476179] LNet: Added LNI 192.168.204.139@tcp18 [8/256/0/180] [19000.039639] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [19000.061721] LNet: Added LNI 192.168.204.139@tcp19 [8/256/0/180] [19005.109897] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [19005.137770] LNet: Added LNI 192.168.204.139@tcp20 [8/256/0/180] [19010.049939] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [19010.083669] LNet: Added LNI 192.168.204.139@tcp21 [8/256/0/180] [19014.746941] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [19014.777327] LNet: Added LNI 192.168.204.139@tcp22 [8/256/0/180] [19019.607534] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [19019.661708] LNet: Added LNI 192.168.204.139@tcp23 [8/256/0/180] [19025.410867] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [19025.455060] LNet: Added LNI 192.168.204.139@tcp24 [8/256/0/180] [19030.540616] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [19030.565948] LNet: Added LNI 192.168.204.139@tcp25 [8/256/0/180] [19035.601075] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [19035.636847] LNet: Added LNI 192.168.204.139@tcp26 [8/256/0/180] [19040.934758] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [19040.976603] LNet: Added LNI 192.168.204.139@tcp27 [8/256/0/180] [19046.423846] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [19046.448424] LNet: Added LNI 192.168.204.139@tcp28 [8/256/0/180] [19052.435454] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [19052.466381] LNet: Added LNI 192.168.204.139@tcp29 [8/256/0/180] [19058.379451] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [19058.412761] LNet: Added LNI 192.168.204.139@tcp30 [8/256/0/180] [19064.739236] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [19064.788964] LNet: Added LNI 192.168.204.139@tcp31 [8/256/0/180] [19070.999970] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [19071.035416] LNet: Added LNI 192.168.204.139@tcp32 [8/256/0/180] [19077.614157] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [19077.668512] LNet: Added LNI 192.168.204.139@tcp33 [8/256/0/180] [19084.778857] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [19084.817233] LNet: Added LNI 192.168.204.139@tcp34 [8/256/0/180] [19091.729583] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [19091.762404] LNet: Added LNI 192.168.204.139@tcp35 [8/256/0/180] [19098.667918] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [19098.714971] LNet: Added LNI 192.168.204.139@tcp36 [8/256/0/180] [19106.001745] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [19106.032578] LNet: Added LNI 192.168.204.139@tcp37 [8/256/0/180] [19113.741938] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [19113.788790] LNet: Added LNI 192.168.204.139@tcp38 [8/256/0/180] [19121.620665] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [19121.651956] LNet: Added LNI 192.168.204.139@tcp39 [8/256/0/180] [19131.077995] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [19131.102360] LNet: Added LNI 192.168.204.139@tcp40 [8/256/0/180] [19139.666215] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [19139.712221] LNet: Added LNI 192.168.204.139@tcp41 [8/256/0/180] [19149.388366] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [19149.423489] LNet: Added LNI 192.168.204.139@tcp42 [8/256/0/180] [19157.627709] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [19157.655051] LNet: Added LNI 192.168.204.139@tcp43 [8/256/0/180] [19165.279761] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [19165.313869] LNet: Added LNI 192.168.204.139@tcp44 [8/256/0/180] [19173.697903] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [19173.724897] LNet: Added LNI 192.168.204.139@tcp45 [8/256/0/180] [19183.022649] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [19183.057838] LNet: Added LNI 192.168.204.139@tcp46 [8/256/0/180] [19191.784549] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [19191.817107] LNet: Added LNI 192.168.204.139@tcp47 [8/256/0/180] [19200.489352] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [19200.518544] LNet: Added LNI 192.168.204.139@tcp48 [8/256/0/180] [19208.879144] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [19208.926105] LNet: Added LNI 192.168.204.139@tcp49 [8/256/0/180] [19218.288389] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [19218.355805] LNet: Added LNI 192.168.204.139@tcp50 [8/256/0/180] [19235.303115] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [19235.317340] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19235.330692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [19238.376552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [19238.380768] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19238.405632] Lustre: Skipped 1 previous similar message [19241.938831] Lustre: server umount lustre-MDT0000 complete [19246.120330] LustreError: 614438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788172768 with bad export cookie 10062468504054123399 [19246.128258] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [19246.139792] LustreError: 614438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [19246.520441] Lustre: server umount lustre-MDT0001 complete [19264.991091] Lustre: 613715:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788172770/real 1788172770] req@00000000f55104c0 x1875034654293760/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788172786 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [19265.019656] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19267.970309] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [19268.065250] Lustre: 613712:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788172773/real 1788172773] req@00000000f6199964 x1875034654293888/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788172789 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [19269.106414] Lustre: 613713:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788172775/real 1788172775] req@000000006b51f7c9 x1875034654294016/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788172791 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [19272.159217] Lustre: 613715:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788172778/real 1788172778] req@00000000defff0b6 x1875034654294272/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788172794 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [19277.343184] Lustre: 613715:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788172783/real 1788172783] req@00000000b2fa6d9e x1875034654294400/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788172799 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [19277.584678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19291.990828] LustreError: 616408:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19292.019130] LustreError: 616408:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [19292.051924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [19294.049414] LustreError: 616409:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19294.076314] LustreError: 616409:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [19295.080323] LustreError: 616408:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19295.089365] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [19297.252480] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [19300.320670] LustreError: 616409:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19305.446581] LustreError: 616408:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19305.890970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19306.214604] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [19309.295604] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [19310.217373] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [19317.689419] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [19324.244408] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [19342.314508] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [19342.325731] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19342.342058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [19342.354199] Lustre: Skipped 2 previous similar messages [19343.330586] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [19345.382933] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [19345.403746] Lustre: Skipped 2 previous similar messages [19348.467271] Lustre: server umount lustre-OST0000 complete [19352.544669] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [19352.560708] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19352.575423] Lustre: Skipped 1 previous similar message [19352.586845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [19358.873297] Lustre: server umount lustre-MDT0000 complete [19360.744311] LustreError: 640657:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19360.780741] LustreError: 640657:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [19363.609856] LustreError: 614438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788172885 with bad export cookie 10062468504054126010 [19363.610947] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [19363.619410] LustreError: 614438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [19364.526921] Lustre: server umount lustre-MDT0001 complete [19378.767811] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [19382.535391] Key type lgssc unregistered [19382.858254] LNet: 643434:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [19382.862897] LNetError: 643434:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [19382.869497] LNet: Removed LNI 192.168.204.139@tcp [19382.871984] LNet: Removed LNI 192.168.204.139@tcp10 [19382.874968] LNet: Removed LNI 192.168.204.139@tcp11 [19382.878972] LNet: Removed LNI 192.168.204.139@tcp12 [19382.886714] LNet: Removed LNI 192.168.204.139@tcp13 [19382.892785] LNet: Removed LNI 192.168.204.139@tcp14 [19382.902600] LNet: Removed LNI 192.168.204.139@tcp15 [19382.910236] LNet: Removed LNI 192.168.204.139@tcp16 [19382.911867] LNet: Removed LNI 192.168.204.139@tcp17 [19382.919118] LNet: Removed LNI 192.168.204.139@tcp18 [19382.920563] LNet: Removed LNI 192.168.204.139@tcp19 [19382.925356] LNet: Removed LNI 192.168.204.139@tcp20 [19382.936286] LNet: Removed LNI 192.168.204.139@tcp21 [19382.948082] LNet: Removed LNI 192.168.204.139@tcp22 [19382.952992] LNet: Removed LNI 192.168.204.139@tcp23 [19382.954404] LNet: Removed LNI 192.168.204.139@tcp24 [19382.957293] LNet: Removed LNI 192.168.204.139@tcp25 [19382.961763] LNet: Removed LNI 192.168.204.139@tcp26 [19382.971780] LNet: Removed LNI 192.168.204.139@tcp27 [19382.974129] LNet: Removed LNI 192.168.204.139@tcp28 [19382.975575] LNet: Removed LNI 192.168.204.139@tcp29 [19382.978606] LNet: Removed LNI 192.168.204.139@tcp30 [19382.981372] LNet: Removed LNI 192.168.204.139@tcp31 [19382.991492] LNet: Removed LNI 192.168.204.139@tcp32 [19382.996619] LNet: Removed LNI 192.168.204.139@tcp33 [19382.999464] LNet: Removed LNI 192.168.204.139@tcp34 [19383.005298] LNet: Removed LNI 192.168.204.139@tcp35 [19383.008343] LNet: Removed LNI 192.168.204.139@tcp36 [19383.015718] LNet: Removed LNI 192.168.204.139@tcp37 [19383.028370] LNet: Removed LNI 192.168.204.139@tcp38 [19383.039955] LNet: Removed LNI 192.168.204.139@tcp39 [19383.042394] LNet: Removed LNI 192.168.204.139@tcp40 [19383.043793] LNet: Removed LNI 192.168.204.139@tcp41 [19383.055136] LNet: Removed LNI 192.168.204.139@tcp42 [19383.066579] LNet: Removed LNI 192.168.204.139@tcp43 [19383.072368] LNet: Removed LNI 192.168.204.139@tcp44 [19383.073911] LNet: Removed LNI 192.168.204.139@tcp45 [19383.081301] LNet: Removed LNI 192.168.204.139@tcp46 [19383.083570] LNet: Removed LNI 192.168.204.139@tcp47 [19383.087055] LNet: Removed LNI 192.168.204.139@tcp48 [19383.093611] LNet: Removed LNI 192.168.204.139@tcp49 [19383.103233] LNet: Removed LNI 192.168.204.139@tcp50 [19383.658212] Key type .llcrypt unregistered [19383.663191] Key type ._llcrypt unregistered [19404.471208] Key type ._llcrypt registered [19404.473019] Key type .llcrypt registered [19404.581965] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_hostid [19420.563241] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [19421.472967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [19421.494453] alg: No test for adler32 (adler32-zlib) [19422.611737] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [19422.864114] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [19424.519485] Key type lgssc registered [19425.694046] Lustre: Echo OBD driver; http://www.lustre.org/ [19464.336832] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [19464.362089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19465.731590] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [19465.862883] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [19465.974314] Lustre: lustre-MDT0000: new disk, initializing [19466.057906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [19466.097216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [19471.110945] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [19483.087672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19483.247668] Lustre: 646631:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [19483.303517] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [19483.309066] Lustre: Skipped 1 previous similar message [19483.421598] Lustre: lustre-MDT0001: new disk, initializing [19483.589120] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [19483.639223] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [19483.659218] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [19488.216052] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [19499.576630] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [19505.588977] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [19512.914589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [19513.166971] Lustre: *** cfs_fail_loc=90f, val=0*** [19513.249547] Lustre: lustre-OST0000: new disk, initializing [19513.253323] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [19513.264252] Lustre: 648648:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [19513.376106] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [19519.482561] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [19521.605622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [19521.632724] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [19521.735822] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [19529.117190] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [19537.384141] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19537.384496] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [19537.415240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [19537.427787] Lustre: Skipped 1 previous similar message [19541.880502] Lustre: server umount lustre-OST0000 complete [19547.620735] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19547.624351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [19547.632391] Lustre: Skipped 2 previous similar messages [19547.648624] Lustre: Skipped 1 previous similar message [19551.654603] Lustre: server umount lustre-MDT0000 complete [19555.392808] LustreError: 646623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788173077 with bad export cookie 5232007614662054220 [19555.394422] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [19555.399775] LustreError: 646623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [19555.705267] Lustre: server umount lustre-MDT0001 complete [19567.321853] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [19570.490979] Key type lgssc unregistered [19570.928928] LNet: 650553:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [19570.941381] LNetError: 650553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [19572.006869] LNet: Removed LNI 192.168.204.139@tcp [19572.847459] Key type .llcrypt unregistered [19572.850414] Key type ._llcrypt unregistered [19593.428904] Key type ._llcrypt registered [19593.431727] Key type .llcrypt registered [19593.595655] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [19595.236689] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [19595.269373] alg: No test for adler32 (adler32-zlib) [19596.418643] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [19596.669444] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [19598.351325] Key type lgssc registered [19599.698654] Lustre: Echo OBD driver; http://www.lustre.org/ [19608.690884] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [19608.725472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19610.476254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [19615.203569] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [19620.833877] LustreError: 651824:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19620.855772] LustreError: 651824:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [19625.602458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19625.987924] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [19630.427156] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [19638.371222] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [19644.336481] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [19651.132425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [19651.692763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [19658.330789] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [19665.762660] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [19679.113254] loop0: detected capacity change from 0 to 204800000 [19679.264828] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [19679.279457] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [19679.298427] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [19685.307911] loop0: detected capacity change from 0 to 204800000 [19685.520105] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [19685.536350] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [19685.548680] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [19690.860454] loop0: detected capacity change from 0 to 204800000 [19697.090269] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [19697.301389] Lustre: fs15246-OST0000: new disk, initializing [19697.305390] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [19697.311420] Lustre: 655420:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [19697.368276] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [19703.783318] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [19714.202802] loop1: detected capacity change from 0 to 204800000 [19720.135333] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19720.280097] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [19720.363865] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [19720.431798] Lustre: fs15246-MDT0000: new disk, initializing [19720.659421] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [19720.696567] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [19720.830864] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [19721.888807] Lustre: 656541:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000e1695de3 x1875035415426688/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788173259 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [19726.539487] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [19731.125410] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [19731.155312] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [19731.219653] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [19749.357971] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19749.376854] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [19754.472164] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [19755.236618] Lustre: server umount fs15246-MDT0000 complete [19759.589957] LustreError: 651820:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19763.150050] Lustre: server umount fs15246-OST0000 complete [19772.899044] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [19772.911192] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19772.922658] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [19773.920199] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [19773.928642] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19774.949464] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [19774.963973] Lustre: Skipped 2 previous similar messages [19778.827908] Lustre: server umount lustre-OST0000 complete [19784.672922] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [19784.681051] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19784.701435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [19788.434316] Lustre: server umount lustre-MDT0000 complete [19790.309400] LustreError: 651825:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19790.325599] LustreError: 651825:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [19791.812198] LustreError: 653666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788173313 with bad export cookie 15951120398812030047 [19791.821217] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [19791.823503] LustreError: 653666:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [19792.219353] Lustre: server umount lustre-MDT0001 complete [19802.171879] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [19804.845628] Key type lgssc unregistered [19805.314299] LNet: 659638:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [19805.325495] LNetError: 659638:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [19806.393907] LNet: Removed LNI 192.168.204.139@tcp [19807.268482] Key type .llcrypt unregistered [19807.271188] Key type ._llcrypt unregistered [19824.177987] Key type ._llcrypt registered [19824.182875] Key type .llcrypt registered [19824.361539] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [19825.409863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [19825.521902] alg: No test for adler32 (adler32-zlib) [19826.549479] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [19826.787142] LNet: Added LNI 192.168.204.139@tcp [8/256/0/180] [19828.463854] Key type lgssc registered [19829.521182] Lustre: Echo OBD driver; http://www.lustre.org/ [19865.365438] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_hostid [19873.535549] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [19907.056896] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [19907.107850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19908.461093] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [19908.499745] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [19908.620463] Lustre: lustre-MDT0000: new disk, initializing [19908.703496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [19908.719078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [19913.591656] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [19923.428418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19923.541244] Lustre: 664105:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [19923.624321] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [19923.632624] Lustre: Skipped 1 previous similar message [19923.730047] Lustre: lustre-MDT0001: new disk, initializing [19923.833726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [19923.879501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [19923.891461] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [19928.985418] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [19937.631510] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [19942.580863] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [19948.874636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [19949.116354] Lustre: lustre-OST0000: new disk, initializing [19949.119844] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [19949.125337] Lustre: 666072:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [19949.229827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [19954.561987] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_default_debug -1 all [19958.873201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [19958.888793] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [19959.012412] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [19964.160050] Lustre: DEBUG MARKER: oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [19968.761184] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [19968.941386] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [19973.311195] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [19973.526704] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [19979.747780] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [19979.749596] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19979.776363] LustreError: Skipped 1 previous similar message [19979.777198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [19979.813025] Lustre: Skipped 1 previous similar message [19981.509851] Lustre: server umount lustre-OST0000 complete [19984.867366] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19984.870679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [19984.879383] Lustre: Skipped 1 previous similar message [19984.889274] Lustre: Skipped 2 previous similar messages [19989.986903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [19989.994567] Lustre: Skipped 1 previous similar message [19995.104709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [19995.116726] Lustre: Skipped 1 previous similar message [19999.199369] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [19999.204118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [19999.217154] Lustre: Skipped 1 previous similar message [19999.358982] Lustre: server umount lustre-MDT0000 complete [20002.569523] LustreError: 664096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788173524 with bad export cookie 18004107458208990564 [20002.571824] LustreError: MGC192.168.204.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [20002.583096] LustreError: 664096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [20002.965479] Lustre: server umount lustre-MDT0001 complete [20028.879311] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing set_hostid [20041.237353] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing load_modules_local [20075.114420] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 19809 sec ========= 06:53:15 (1788173595) [20076.929580] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 06:53:17 (1788173597) === [20081.052170] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 06:53:21 (1788173601) === [20122.520265] Lustre: DEBUG MARKER: oleg439-server.virtnet: executing unload_modules_local [20125.452987] Key type lgssc unregistered [20125.778415] LNet: 672716:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [20125.784074] LNetError: 672716:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [20125.800788] LNet: Removed LNI 192.168.204.139@tcp [20126.950438] Key type .llcrypt unregistered [20126.953411] Key type ._llcrypt unregistered