[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 494498844 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.988 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a2000-0x1465ccfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002386] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.007734] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22982c12b6d, max_idle_ns: 440795281273 ns [ 0.008022] Calibrating delay loop (skipped) preset value.. 4799.97 BogoMIPS (lpj=2399988) [ 0.009014] pid_max: default: 32768 minimum: 301 [ 0.010132] LSM: Security Framework initializing [ 0.011063] Yama: becoming mindful. [ 0.012036] SELinux: Initializing. [ 0.013090] *** VALIDATE selinux *** [ 0.022165] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027240] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028153] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029125] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030112] *** VALIDATE tmpfs *** [ 0.031440] *** VALIDATE proc *** [ 0.033077] *** VALIDATE cgroup *** [ 0.034008] *** VALIDATE cgroup2 *** [ 0.035274] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036127] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038030] Spectre V2 : User space: Vulnerable [ 0.039010] Speculative Store Bypass: Vulnerable [ 0.041932] debug: unmapping init [mem 0xffffffffbbe59000-0xffffffffbbe60fff] [ 0.044000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044696] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045024] ... version: 2 [ 0.046014] ... bit width: 48 [ 0.047014] ... generic registers: 4 [ 0.048013] ... value mask: 0000ffffffffffff [ 0.049015] ... max period: 00007fffffffffff [ 0.050015] ... fixed-purpose events: 3 [ 0.051014] ... event mask: 000000070000000f [ 0.052342] rcu: Hierarchical SRCU implementation. [ 0.054489] smp: Bringing up secondary CPUs ... [ 0.055604] x86: Booting SMP configuration: [ 0.056031] .... node #0, CPUs: #1 #2 #3 [ 0.060074] smp: Brought up 1 node, 4 CPUs [ 0.062015] smpboot: Max logical packages: 1 [ 0.063019] smpboot: Total of 4 processors activated (19199.90 BogoMIPS) [ 0.139486] node 0 deferred pages initialised in 71ms [ 0.142626] devtmpfs: initialized [ 0.144278] x86/mm: Memory block size: 128MB [ 0.147655] gcov: version magic: 0x41383552 [ 0.151326] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.152082] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.153265] pinctrl core: initialized pinctrl subsystem [ 0.154162] [ 0.154533] ************************************************************* [ 0.155010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.156011] ** ** [ 0.157014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.158012] ** ** [ 0.159015] ** This means that this kernel is built to expose internal ** [ 0.160012] ** IOMMU data structures, which may compromise security on ** [ 0.161014] ** your system. ** [ 0.162014] ** ** [ 0.163015] ** If you see this message and you are not debugging the ** [ 0.164016] ** kernel, report this immediately to your vendor! ** [ 0.165048] ** ** [ 0.166015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.167017] ************************************************************* [ 0.168636] NET: Registered protocol family 16 [ 0.169430] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.170063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.171065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.173023] cpuidle: using governor menu [ 0.175029] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.178460] PCI: Using configuration type 1 for base access [ 0.180124] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.190051] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.192038] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.196159] cryptd: max_cpu_qlen set to 1000 [ 0.199324] ACPI: Added _OSI(Module Device) [ 0.201017] ACPI: Added _OSI(Processor Device) [ 0.203015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.205060] ACPI: Added _OSI(Processor Aggregator Device) [ 0.210053] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.216510] ACPI: Interpreter enabled [ 0.217051] ACPI: PM: (supports S0 S3 S4 S5) [ 0.218013] ACPI: Using IOAPIC for interrupt routing [ 0.219143] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.220481] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.229087] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.231047] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.234020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.237082] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.241000] acpiphp: Slot [2] registered [ 0.243124] acpiphp: Slot [5] registered [ 0.245102] acpiphp: Slot [6] registered [ 0.246138] acpiphp: Slot [7] registered [ 0.248116] acpiphp: Slot [8] registered [ 0.250139] acpiphp: Slot [9] registered [ 0.251133] acpiphp: Slot [10] registered [ 0.253131] acpiphp: Slot [3] registered [ 0.254128] acpiphp: Slot [4] registered [ 0.256117] acpiphp: Slot [11] registered [ 0.257109] acpiphp: Slot [12] registered [ 0.259116] acpiphp: Slot [13] registered [ 0.261114] acpiphp: Slot [14] registered [ 0.262108] acpiphp: Slot [15] registered [ 0.264125] acpiphp: Slot [16] registered [ 0.265128] acpiphp: Slot [17] registered [ 0.267118] acpiphp: Slot [18] registered [ 0.269111] acpiphp: Slot [19] registered [ 0.270113] acpiphp: Slot [20] registered [ 0.272134] acpiphp: Slot [21] registered [ 0.273110] acpiphp: Slot [22] registered [ 0.275136] acpiphp: Slot [23] registered [ 0.276104] acpiphp: Slot [24] registered [ 0.278101] acpiphp: Slot [25] registered [ 0.279104] acpiphp: Slot [26] registered [ 0.281138] acpiphp: Slot [27] registered [ 0.282107] acpiphp: Slot [28] registered [ 0.284137] acpiphp: Slot [29] registered [ 0.285107] acpiphp: Slot [30] registered [ 0.287114] acpiphp: Slot [31] registered [ 0.289085] PCI host bridge to bus 0000:00 [ 0.290019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.292026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.295030] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.298031] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.300026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.303029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.305184] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.307968] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.311250] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.320831] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.327057] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.330021] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.332013] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.333021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.336063] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.338794] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.342052] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.344818] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.350014] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.362975] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.366948] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.373260] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.379016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.385015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.409015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.418646] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.424017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.428022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.443017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.451883] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.457015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.465017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.478025] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.488469] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.497015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.505016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.531020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.541088] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.549014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.556016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.576018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.585241] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.593016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.602016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.619018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.633000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.636422] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.638381] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.641378] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.643243] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.649122] iommu: Default domain type: Passthrough [ 0.651446] SCSI subsystem initialized [ 0.653177] ACPI: bus type USB registered [ 0.654132] usbcore: registered new interface driver usbfs [ 0.656098] usbcore: registered new interface driver hub [ 0.658078] usbcore: registered new device driver usb [ 0.660183] pps_core: LinuxPPS API ver. 1 registered [ 0.662012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.665059] PTP clock support registered [ 0.668077] EDAC MC: Ver: 3.0.0 [ 0.670187] PCI: Using ACPI for IRQ routing [ 0.671842] NetLabel: Initializing [ 0.672000] NetLabel: domain hash size = 128 [ 0.674018] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.676078] NetLabel: unlabeled traffic allowed by default [ 0.679082] vgaarb: loaded [ 0.680275] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.682015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.687005] clocksource: Switched to clocksource kvm-clock [ 0.797235] VFS: Disk quotas dquot_6.6.0 [ 0.799077] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.801526] *** VALIDATE ramfs *** [ 0.803049] *** VALIDATE hugetlbfs *** [ 0.804876] pnp: PnP ACPI init [ 0.808161] pnp: PnP ACPI: found 6 devices [ 0.823733] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.827255] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.829767] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.832008] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.834420] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.837034] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.840101] NET: Registered protocol family 2 [ 0.842706] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.848241] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.852840] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.858960] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.862944] TCP: Hash tables configured (established 65536 bind 65536) [ 0.866103] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.868987] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.871880] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.875069] NET: Registered protocol family 1 [ 0.877924] RPC: Registered named UNIX socket transport module. [ 0.880154] RPC: Registered udp transport module. [ 0.881839] RPC: Registered tcp transport module. [ 0.883593] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.886054] NET: Registered protocol family 44 [ 0.887770] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.890225] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.892507] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.894957] PCI: CLS 0 bytes, default 64 [ 0.897378] Unpacking initramfs... [ 2.337291] debug: unmapping init [mem 0xffff9fffbcc54000-0xffff9fffbffbffff] [ 2.343816] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.346936] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.350709] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22982c12b6d, max_idle_ns: 440795281273 ns [ 2.860121] Initialise system trusted keyrings [ 2.861903] Key type blacklist registered [ 2.864076] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.873404] zbud: loaded [ 2.876526] *** VALIDATE nfs *** [ 2.877911] *** VALIDATE nfs4 *** [ 2.879793] pstore: using deflate compression [ 2.886731] Platform Keyring initialized [ 2.991523] NET: Registered protocol family 38 [ 2.993540] Key type asymmetric registered [ 2.995254] Asymmetric key parser 'x509' registered [ 2.997122] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.000144] io scheduler mq-deadline registered [ 3.002317] io scheduler kyber registered [ 3.004414] io scheduler bfq registered [ 3.007135] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.010189] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.013110] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.015876] ACPI: Power Button [PWRF] [ 3.020619] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.025926] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.039491] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.045239] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.062553] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.091945] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.121304] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.127454] Non-volatile memory driver v1.3 [ 3.128894] Linux agpgart interface v0.103 [ 3.161404] virtio_blk virtio1: [vda] 139480 512-byte logical blocks (71.4 MB/68.1 MiB) [ 3.164760] vda: detected capacity change from 0 to 71413760 [ 3.179256] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.181621] vdb: detected capacity change from 0 to 1073741824 [ 3.198506] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.201535] vdc: detected capacity change from 0 to 2621440000 [ 3.215678] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.218809] vdd: detected capacity change from 0 to 2621440000 [ 3.236353] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.239666] vde: detected capacity change from 0 to 4294967296 [ 3.256815] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.259834] vdf: detected capacity change from 0 to 4294967296 [ 3.265527] libphy: Fixed MDIO Bus: probed [ 3.270616] usbcore: registered new interface driver usbserial_generic [ 3.272566] usbserial: USB Serial support registered for generic [ 3.274836] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.280357] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.283089] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.286407] mousedev: PS/2 mouse device common for all mice [ 3.289535] rtc_cmos 00:05: RTC can wake from S4 [ 3.292689] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.294940] rtc_cmos 00:05: registered as rtc0 [ 3.304227] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.304983] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.308764] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.311593] intel_pstate: CPU model not supported [ 3.316974] hid: raw HID events driver (C) Jiri Kosina [ 3.319875] usbcore: registered new interface driver usbhid [ 3.323037] usbhid: USB HID core driver [ 3.325405] drop_monitor: Initializing network drop monitor service [ 3.327604] Initializing XFRM netlink socket [ 3.329484] NET: Registered protocol family 10 [ 3.332648] Segment Routing with IPv6 [ 3.334266] NET: Registered protocol family 17 [ 3.336608] mpls_gso: MPLS GSO support [ 3.341483] RAS: Correctable Errors collector initialized. [ 3.343410] AVX version of gcm_enc/dec engaged. [ 3.344717] AES CTR mode by8 optimization enabled [ 3.432973] sched_clock: Marking stable (3432937944, 0)->(4387064565, -954126621) [ 3.436385] registered taskstats version 1 [ 3.438389] Loading compiled-in X.509 certificates [ 3.440756] zswap: loaded using pool lzo/zbud [ 3.466324] Key type big_key registered [ 3.480421] Key type encrypted registered [ 3.482023] ima: No TPM chip found, activating TPM-bypass! [ 3.484197] ima: Allocated hash algorithm: sha1 [ 3.485741] ima: No architecture policies found [ 3.487467] evm: Initialising EVM extended attributes: [ 3.489434] evm: security.selinux [ 3.490623] evm: security.ima [ 3.491778] evm: security.capability [ 3.493138] evm: HMAC attrs: 0x1 [ 3.495449] rtc_cmos 00:05: setting system clock to 2026-06-12 13:58:41 UTC (1781272721) [ 3.501176] debug: unmapping init [mem 0xffffffffbce03000-0xffffffffbcffffff] [ 3.504342] debug: unmapping init [mem 0xffffffffbbb82000-0xffffffffbbe58fff] [ 3.517091] Write protecting the kernel read-only data: 28672k [ 3.520783] debug: unmapping init [mem 0xffffffffba203000-0xffffffffba3fffff] [ 3.523215] debug: unmapping init [mem 0xffffffffbab14000-0xffffffffbabfffff] [ 3.556933] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.563432] systemd[1]: Detected virtualization kvm. [ 3.564903] systemd[1]: Detected architecture x86-64. [ 3.566642] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.590272] systemd[1]: No hostname configured. [ 3.591962] systemd[1]: Set hostname to . [ 3.593807] random: systemd: uninitialized urandom read (16 bytes read) [ 3.596106] systemd[1]: Initializing machine ID from random generator. [ 3.640362] random: ln: uninitialized urandom read (6 bytes read) [ 3.743222] random: systemd: uninitialized urandom read (16 bytes read) [ 3.747849] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.753040] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.760353] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Swap. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.476600] device-mapper: uevent: version 1.0.3 [ 4.479119] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.262618] virtio_net virtio0 ens2: renamed from eth0 [ 5.274276] random: fast init done [ 5.334405] scsi host0: ata_piix [ 5.402804] scsi host1: ata_piix [ 5.404591] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.407293] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.078135] dracut-initqueue[593]: RTNETLINK answers: File exists [ 9.998036] random: crng init done [ 9.999370] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.523706] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.751689] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.077640] SELinux: Disabled at runtime. [ 12.147572] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.162932] systemd[1]: Detected virtualization kvm. [ 12.165643] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.712722] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.716758] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.721737] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.725356] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.728383] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.740269] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.745357] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on udev Kernel Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting Huge Pages File System... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. Mounting POSIX Message Queue File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Mounting Kernel Debug File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting udev Coldplug all Devices... [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. [ 12.935370] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.201705] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.446245] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.461421] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.583479] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.599668] EDAC sbridge: Ver: 1.1.2 [ 15.206477] Key type dns_resolver registered [ 15.503717] NFS: Registering the id_resolver key type [ 15.505595] Key type id_resolver registered [ 15.506695] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started dnf makecache --timer. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg459-server login: [ 41.557454] libcfs: loading out-of-tree module taints kernel. [ 41.579349] Key type ._llcrypt registered [ 41.580994] Key type .llcrypt registered [ 41.638239] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 62.784123] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 64.827337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 64.856165] alg: No test for adler32 (adler32-zlib) [ 66.295854] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 67.155853] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 68.920263] Key type lgssc registered [ 71.253915] Lustre: Echo OBD driver; http://www.lustre.org/ [ 85.514050] hrtimer: interrupt took 3043242 ns [ 92.022410] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 137.648538] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 149.220669] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 149.254251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 150.507346] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 150.584912] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 150.754398] Lustre: lustre-MDT0000: new disk, initializing [ 150.896076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 150.922769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 155.787237] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 170.437369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 170.553808] Lustre: 6527:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 170.602578] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 170.611673] Lustre: Skipped 1 previous similar message [ 170.760310] Lustre: lustre-MDT0001: new disk, initializing [ 170.824066] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 170.856409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 170.868934] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 175.230103] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 180.394825] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 189.843690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 190.216619] Lustre: lustre-OST0000: new disk, initializing [ 190.231776] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 190.241049] Lustre: 8465:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 190.301474] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 195.077068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 195.096864] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 195.192273] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 197.143146] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 212.352488] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 212.550976] Lustre: lustre-OST0001: new disk, initializing [ 212.555878] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 212.562712] Lustre: 9539:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 212.658103] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 219.215298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 219.235285] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 219.337449] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 219.939926] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 231.840097] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 242.191803] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 248.157232] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing check_logdir /tmp/testlogs/ [ 254.246846] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing yml_node [ 258.676387] Lustre: DEBUG MARKER: Client: 2.17.54.4 [ 261.130483] Lustre: DEBUG MARKER: MDS: 2.17.54.4 [ 263.441551] Lustre: DEBUG MARKER: OSS: 2.17.54.4 [ 265.192610] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Jun 12 10:03:01 EDT 2026 [ 279.882750] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 281.431348] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 298.980866] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 298.995522] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 299.023588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 301.537228] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 301.539090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 301.553910] Lustre: Skipped 1 previous similar message [ 301.561179] Lustre: Skipped 1 previous similar message [ 303.362780] Lustre: server umount lustre-MDT0000 complete [ 307.090308] LustreError: 8464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781273025 with bad export cookie 82144960876890238 [ 307.097737] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 307.117207] LustreError: 8464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 311.777425] LustreError: 9163:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 311.779441] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 311.783516] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 311.783524] Lustre: Skipped 2 previous similar messages [ 311.798239] LustreError: 9163:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 311.812949] Lustre: Skipped 2 previous similar messages [ 312.802342] LustreError: 6535:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 312.830421] LustreError: 6535:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 313.493325] Lustre: server umount lustre-MDT0001 complete [ 317.663559] Lustre: server umount lustre-OST0000 complete [ 321.796592] Lustre: server umount lustre-OST0001 complete [ 328.956345] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 339.244506] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 374.392190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 374.655751] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 374.684994] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 374.799912] Lustre: lustre-MDT0000: new disk, initializing [ 374.885613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 374.901256] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 379.808202] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 390.054840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 390.156697] Lustre: 15094:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 390.223965] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 390.228313] Lustre: Skipped 1 previous similar message [ 390.314241] Lustre: lustre-MDT0001: new disk, initializing [ 390.412383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 390.479027] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 390.491266] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 395.185219] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 405.238814] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 411.090973] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 418.049581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 418.326966] Lustre: lustre-OST0000: new disk, initializing [ 418.331600] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 418.340486] Lustre: 17060:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 418.464150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 420.454498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 420.462741] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 420.629260] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 425.719970] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 435.759556] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 441.653723] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 441.945660] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 448.420134] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 448.786558] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 456.675222] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 456.696045] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 456.711028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 456.721521] Lustre: Skipped 4 previous similar messages [ 457.875928] Lustre: server umount lustre-OST0000 complete [ 461.798833] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 461.800206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 461.812364] Lustre: Skipped 2 previous similar messages [ 461.830363] Lustre: Skipped 1 previous similar message [ 472.045764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 472.060238] Lustre: Skipped 3 previous similar messages [ 476.128202] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 476.244606] LustreError: 18600:0:(ldlm_resource.c:1179:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (ffffa0003584a500) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 476.376758] Lustre: server umount lustre-MDT0000 complete [ 480.646530] LustreError: 15086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781273198 with bad export cookie 82144960876891771 [ 480.655830] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 480.672247] LustreError: 15086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 480.966121] Lustre: server umount lustre-MDT0001 complete [ 485.512241] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 10:06:41 (1781273201) [ 488.030308] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 489.836937] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 10:06:45 (1781273205) [ 497.142973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 497.603310] LustreError: 19472:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 497.687658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 502.778477] LustreError: 19473:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 504.788555] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 507.874342] LustreError: 19472:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 514.902462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 519.829715] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 528.361250] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 534.481421] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 541.699707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 549.575646] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 558.254797] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 568.801421] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 568.810581] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 568.835898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 568.851903] Lustre: Skipped 2 previous similar messages [ 570.849684] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 574.906881] Lustre: server umount lustre-OST0000 complete [ 587.132476] Lustre: server umount lustre-MDT0000 complete [ 590.307082] LustreError: 19472:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 590.322462] LustreError: 19472:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 591.392890] LustreError: 22053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781273309 with bad export cookie 82144960876892653 [ 591.407520] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 591.408675] LustreError: 22053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 591.808665] Lustre: server umount lustre-MDT0001 complete [ 604.332151] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 607.702448] Key type lgssc unregistered [ 608.128153] LNet: 23008:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 608.135037] LNetError: 23008:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 608.151813] LNet: Removed LNI 192.168.204.159@tcp [ 609.268162] Key type .llcrypt unregistered [ 609.269743] Key type ._llcrypt unregistered [ 632.126253] Key type ._llcrypt registered [ 632.128959] Key type .llcrypt registered [ 632.250541] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 633.301603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 633.324964] alg: No test for adler32 (adler32-zlib) [ 634.300042] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 634.506097] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 636.177053] Key type lgssc registered [ 637.300904] Lustre: Echo OBD driver; http://www.lustre.org/ [ 645.116888] loop: module loaded [ 645.182264] loop0: detected capacity change from 0 to 204800000 [ 645.373070] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 645.400527] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 645.845252] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 651.322155] loop0: detected capacity change from 0 to 204800000 [ 651.474475] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 651.485879] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 651.493548] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 656.499774] loop0: detected capacity change from 0 to 204800000 [ 661.622062] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 661.654631] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 663.013404] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 663.068859] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 663.134961] Lustre: lustre-MDT0000: new disk, initializing [ 663.238731] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 663.267656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 667.444639] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 676.471501] loop1: detected capacity change from 0 to 204800000 [ 682.383492] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 682.752338] Lustre: lustre-OST07c6: new disk, initializing [ 682.757083] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 682.764098] Lustre: Skipped 1 previous similar message [ 682.778487] Lustre: 26194:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 682.920061] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 688.260553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 688.268473] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 688.311900] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 689.578070] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 694.452517] Lustre: 26478:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 255 < left 264, rollback = 2 [ 694.468167] Lustre: 26478:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 694.476596] Lustre: 26478:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 694.483408] Lustre: 26478:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 694.495774] Lustre: 26478:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 694.508355] Lustre: 26478:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 715.144916] Lustre: 26478:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 715.161185] Lustre: 26478:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 715.169315] Lustre: 26478:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 715.174239] Lustre: 26478:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 715.187485] Lustre: 26478:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 715.199945] Lustre: 26478:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 720.144787] Lustre: 25179:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 720.161612] Lustre: 25179:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 720.181325] Lustre: 25179:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 720.191765] Lustre: 25179:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 720.200293] Lustre: 25179:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 720.209674] Lustre: 25179:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 720.216905] Lustre: 25179:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 720.224222] Lustre: 25179:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 720.238117] Lustre: 25179:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 720.248737] Lustre: 25179:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 720.268894] Lustre: 25179:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 720.278309] Lustre: 25179:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 723.516627] Lustre: 26478:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 723.534129] Lustre: 26478:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 723.545855] Lustre: 26478:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 723.556727] Lustre: 26478:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 723.570229] Lustre: 26478:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 723.579892] Lustre: 26478:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 723.588901] Lustre: 26478:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 723.600350] Lustre: 26478:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 723.612263] Lustre: 26478:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 723.622434] Lustre: 26478:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 723.628258] Lustre: 26478:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 723.641293] Lustre: 26478:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 728.349414] Lustre: 25179:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 728.362546] Lustre: 25179:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 728.371533] Lustre: 25179:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 728.380305] Lustre: 25179:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 728.387429] Lustre: 25179:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 728.393428] Lustre: 25179:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 728.399095] Lustre: 25179:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 728.405480] Lustre: 25179:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 728.410176] Lustre: 25179:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 728.414316] Lustre: 25179:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 728.419381] Lustre: 25179:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 728.426092] Lustre: 25179:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 739.559210] Lustre: 25179:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 739.567548] Lustre: 25179:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 739.572285] Lustre: 25179:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 739.579203] Lustre: 25179:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 739.589456] Lustre: 25179:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 739.600251] Lustre: 25179:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 739.610600] Lustre: 25179:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 739.617497] Lustre: 25179:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 739.625303] Lustre: 25179:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 739.635147] Lustre: 25179:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 739.645235] Lustre: 25179:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 739.653707] Lustre: 25179:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 790.498275] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 790.523057] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 792.516993] Lustre: server umount lustre-OST07c6 complete [ 802.460413] Lustre: server umount lustre-MDT0000 complete [ 819.986290] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 823.000182] Key type lgssc unregistered [ 823.327234] LNet: 28758:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 823.334218] LNetError: 28758:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 823.360318] LNet: Removed LNI 192.168.204.159@tcp [ 824.211677] Key type .llcrypt unregistered [ 824.215283] Key type ._llcrypt unregistered [ 842.559180] Key type ._llcrypt registered [ 842.560718] Key type .llcrypt registered [ 842.650528] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 843.798700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 843.835526] alg: No test for adler32 (adler32-zlib) [ 844.876781] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 845.086538] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 846.744216] Key type lgssc registered [ 847.689815] Lustre: Echo OBD driver; http://www.lustre.org/ [ 855.394205] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 855.414852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 856.888140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 861.383349] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 866.303597] LustreError: 30027:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 866.327226] LustreError: 30027:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 869.380942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 869.781435] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 874.891533] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 883.023440] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 888.734171] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 896.729716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 897.256896] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 901.374862] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 904.504293] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 912.165751] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 932.324831] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 932.335851] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 932.361463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 933.293761] Lustre: server umount lustre-OST0000 complete [ 940.518523] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 940.524741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 943.675838] Lustre: server umount lustre-MDT0000 complete [ 945.639736] LustreError: 30027:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 945.660775] LustreError: 30027:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 947.824356] LustreError: 30008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781273665 with bad export cookie 3542736308157266192 [ 947.833904] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 947.846190] LustreError: 30008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 957.580797] Lustre: server umount lustre-MDT0001 complete [ 974.167348] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 978.696668] Key type lgssc unregistered [ 979.343731] LNet: 33768:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 979.353499] LNetError: 33768:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 979.373790] LNet: Removed LNI 192.168.204.159@tcp [ 981.311156] Key type .llcrypt unregistered [ 981.314248] Key type ._llcrypt unregistered [ 1006.863607] Key type ._llcrypt registered [ 1006.865968] Key type .llcrypt registered [ 1006.987472] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 1008.663623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1008.776155] alg: No test for adler32 (adler32-zlib) [ 1009.995669] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 1010.394841] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 1012.153459] Key type lgssc registered [ 1013.718988] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1021.932331] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1021.948196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1023.506906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1028.560180] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1033.698610] LustreError: 35042:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1033.728726] LustreError: 35042:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1036.533358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1036.842616] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1041.337898] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1049.128774] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1054.692313] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1061.734045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1062.138545] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1068.302756] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1068.947506] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1077.179241] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1089.003652] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1089.004279] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1089.027105] LustreError: Skipped 1 previous similar message [ 1089.058153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1089.076775] Lustre: Skipped 1 previous similar message [ 1094.119730] LustreError: 36867:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1094.274515] Lustre: server umount lustre-OST0000 complete [ 1099.235166] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1099.247561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1099.260812] Lustre: Skipped 2 previous similar messages [ 1104.353883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1104.366203] Lustre: Skipped 2 previous similar messages [ 1104.870509] Lustre: server umount lustre-MDT0000 complete [ 1109.121471] LustreError: 35780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781273827 with bad export cookie 16240965766943213173 [ 1109.122051] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1109.129121] LustreError: 35780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1109.413515] Lustre: server umount lustre-MDT0001 complete [ 1122.036389] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 1125.175959] Key type lgssc unregistered [ 1125.559450] LNet: 38578:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1125.571973] LNetError: 38578:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1125.590093] LNet: Removed LNI 192.168.204.159@tcp [ 1126.609129] Key type .llcrypt unregistered [ 1126.611885] Key type ._llcrypt unregistered [ 1152.698869] Key type ._llcrypt registered [ 1152.700952] Key type .llcrypt registered [ 1152.823510] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 1154.191071] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1154.215169] alg: No test for adler32 (adler32-zlib) [ 1155.367401] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 1155.669874] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 1157.354056] Key type lgssc registered [ 1158.843783] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1168.000243] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1168.035442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1169.584608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1175.448539] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1179.125496] LustreError: 39995:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1179.157214] LustreError: 39995:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1184.228512] LustreError: 39994:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1185.457814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1185.856890] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1190.538375] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1199.726686] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1207.004522] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1215.234986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1215.793942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1223.039353] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1232.864352] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1239.365523] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1244.645674] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1244.649390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1244.663247] Lustre: Skipped 2 previous similar messages [ 1244.689801] Lustre: Skipped 2 previous similar messages [ 1246.783439] Lustre: server umount lustre-MDT0000 complete [ 1249.789813] LustreError: 39994:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1249.819040] LustreError: 39994:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1251.113929] LustreError: 39977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781273969 with bad export cookie 8009962809194366416 [ 1251.150876] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1254.886578] LustreError: 42575:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1254.888143] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1254.911877] LustreError: 42575:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1254.948439] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1254.953273] Lustre: Skipped 1 previous similar message [ 1256.931145] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1256.942621] Lustre: Skipped 1 previous similar message [ 1257.649421] Lustre: server umount lustre-MDT0001 complete [ 1280.182385] Lustre: server umount lustre-OST0000 complete [ 1298.315555] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 1301.398962] Key type lgssc unregistered [ 1301.820070] LNet: 43857:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1301.839112] LNetError: 43857:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1301.866193] LNet: Removed LNI 192.168.204.159@tcp [ 1303.024301] Key type .llcrypt unregistered [ 1303.025889] Key type ._llcrypt unregistered [ 1331.424228] Key type ._llcrypt registered [ 1331.428390] Key type .llcrypt registered [ 1331.525560] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 1332.707745] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1332.717853] alg: No test for adler32 (adler32-zlib) [ 1334.061240] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 1334.374761] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 1336.256170] Key type lgssc registered [ 1337.982243] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1348.939902] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1348.953194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1350.676194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1357.155551] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1360.882261] LustreError: 45275:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1360.913550] LustreError: 45275:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1366.008786] LustreError: 45274:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1367.972613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1368.562155] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1374.704946] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1382.736261] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1389.062639] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1395.838615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1396.092404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1403.173690] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1411.923587] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1420.648161] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1428.136995] Lustre: lustre-OST0000: Not available for connect from 192.168.204.59@tcp (stopping) [ 1428.963773] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1428.979512] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1429.005626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1431.013590] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1431.036553] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1431.041597] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1433.234786] Lustre: lustre-OST0000: Not available for connect from 192.168.204.59@tcp (stopping) [ 1433.257161] Lustre: Skipped 1 previous similar message [ 1434.232230] Lustre: server umount lustre-OST0000 complete [ 1455.589262] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1455.602201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1455.632285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1460.943627] Lustre: server umount lustre-MDT0000 complete [ 1462.766327] LustreError: 47117:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1462.792556] LustreError: 47117:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1465.108729] LustreError: 45254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781274183 with bad export cookie 8980782303762958057 [ 1465.119204] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1465.122736] LustreError: 45254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1471.518979] Lustre: server umount lustre-MDT0001 complete [ 1485.069750] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 1489.333315] Key type lgssc unregistered [ 1489.716852] LNet: 49011:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1489.724758] LNetError: 49011:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1489.738375] LNet: Removed LNI 192.168.204.159@tcp [ 1490.644520] Key type .llcrypt unregistered [ 1490.649876] Key type ._llcrypt unregistered [ 1518.993445] Key type ._llcrypt registered [ 1518.997387] Key type .llcrypt registered [ 1519.150934] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 1520.403722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1520.563091] alg: No test for adler32 (adler32-zlib) [ 1521.794784] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 1522.215217] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 1524.024275] Key type lgssc registered [ 1525.392158] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1534.552465] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1534.603465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1536.143278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1541.327235] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1546.731248] LustreError: 50430:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1546.762895] LustreError: 50430:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1551.165623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1551.829054] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1557.628718] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1567.223451] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1573.795665] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1581.535725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1582.212969] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1589.309556] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1598.035550] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1606.776367] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1613.856536] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1617.646750] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1630.494692] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-06-1210h25m45s [ 1632.710764] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1637.862608] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1637.879437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1637.889620] Lustre: Skipped 1 previous similar message [ 1637.902237] Lustre: Skipped 1 previous similar message [ 1638.947976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1638.957369] Lustre: Skipped 1 previous similar message [ 1640.664095] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.59@tcp (stopping) [ 1640.677380] Lustre: Skipped 3 previous similar messages [ 1641.677800] Lustre: server umount lustre-MDT0000 complete [ 1645.029393] LustreError: 50430:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1645.056388] LustreError: 50430:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1646.752087] LustreError: 51670:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1647.106713] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1651.171680] LustreError: 52547:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1651.185232] LustreError: 52547:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1655.432331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1655.631962] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1655.820801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1655.873408] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1656.902737] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1661.376334] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1665.985285] Lustre: DEBUG MARKER: Wait for df (36540) ... [ 1667.839872] Lustre: DEBUG MARKER: done [ 1673.700820] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1673.706800] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1673.732271] Lustre: Skipped 1 previous similar message [ 1673.742126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1676.257899] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1676.278763] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1682.406266] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1682.411877] Lustre: Skipped 4 previous similar messages [ 1687.009717] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1687.283731] Lustre: server umount lustre-OST0000 complete [ 1692.643637] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1692.660967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1697.454903] Lustre: server umount lustre-MDT0000 complete [ 1699.814970] LustreError: 50429:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1699.838340] LustreError: 50429:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1701.842277] LustreError: 50409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781274419 with bad export cookie 9965435759874061141 [ 1701.842944] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1701.865742] LustreError: 50409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1702.363973] Lustre: server umount lustre-MDT0001 complete [ 1715.903427] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 1719.471797] Key type lgssc unregistered [ 1719.804885] LNet: 55392:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1719.821961] LNetError: 55392:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1719.852883] LNet: Removed LNI 192.168.204.159@tcp [ 1721.075572] Key type .llcrypt unregistered [ 1721.079981] Key type ._llcrypt unregistered [ 1765.781981] Key type ._llcrypt registered [ 1765.783971] Key type .llcrypt registered [ 1765.927772] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 1767.233742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1767.346404] alg: No test for adler32 (adler32-zlib) [ 1768.593944] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 1769.031553] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 1770.760202] Key type lgssc registered [ 1772.298877] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1782.911380] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1782.933982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1784.293473] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1784.303903] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1784.465866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1790.192935] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1800.790877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1800.945966] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1800.992803] Lustre: 57600:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1801.403727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1806.590063] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1814.430117] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1820.109425] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1827.052645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1827.328378] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1827.570248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1834.258110] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 1837.205284] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1841.864848] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1849.658957] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-06-1210h29m25s [ 1851.310378] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1861.131630] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1864.070686] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1864.148309] LustreError: 59941:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1884.358826] Lustre: lustre-MDT0000: Client 8b332abe-beb3-485a-a8e9-cf0e5d543a06 (at 192.168.204.59@tcp) reconnecting [ 1904.168078] LustreError: 59941:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1906.184347] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1908.916169] Lustre: DEBUG MARKER: done [ 1918.945598] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1918.957637] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1918.970854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1919.459945] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1919.495953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1919.506976] Lustre: Skipped 1 previous similar message [ 1922.283899] Lustre: server umount lustre-OST0000 complete [ 1929.697925] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1929.699237] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1929.721672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1929.731831] Lustre: Skipped 1 previous similar message [ 1932.330628] Lustre: server umount lustre-MDT0000 complete [ 1936.094841] LustreError: 57592:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781274654 with bad export cookie 3618409766433485736 [ 1936.107292] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1936.482820] Lustre: server umount lustre-MDT0001 complete [ 1948.280828] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 1951.045580] Key type lgssc unregistered [ 1951.355719] LNet: 61934:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1951.380274] LNetError: 61934:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1952.424476] LNet: Removed LNI 192.168.204.159@tcp [ 1953.186612] Key type .llcrypt unregistered [ 1953.189601] Key type ._llcrypt unregistered [ 1996.825362] Key type ._llcrypt registered [ 1996.830222] Key type .llcrypt registered [ 1996.980410] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 1997.911380] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1997.925195] alg: No test for adler32 (adler32-zlib) [ 1998.985341] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 1999.299458] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 2000.968222] Key type lgssc registered [ 2002.354190] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2005.814290] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2005.841370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2007.109365] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2007.152255] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2007.375370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2008.827474] Lustre: Failing over lustre-MDT0000 [ 2009.432192] Lustre: server umount lustre-MDT0000 complete [ 2016.921622] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 10:32:13 (1781274733) [ 2024.326808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2024.722903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2029.789709] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2038.745122] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2038.873065] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2038.922189] Lustre: 64739:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2039.259177] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2043.911728] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2051.496842] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2056.907506] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2063.547566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2063.773732] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2063.984549] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2070.775487] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2071.076283] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2077.496706] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2083.632607] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2090.466297] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2090.481468] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2090.499412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2091.491770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2091.498859] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2091.505879] Lustre: Skipped 1 previous similar message [ 2091.521764] Lustre: Skipped 1 previous similar message [ 2098.323239] Lustre: server umount lustre-MDT0000 complete [ 2101.720519] LustreError: 64730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781274819 with bad export cookie 3910754386580717634 [ 2101.722961] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2101.734259] LustreError: 64730:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2101.752198] LustreError: 64745:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2101.759774] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2101.770456] LustreError: 64745:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2101.790179] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2101.797196] Lustre: Skipped 1 previous similar message [ 2102.052785] Lustre: server umount lustre-MDT0001 complete [ 2105.778988] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2110.941792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2120.789443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2126.694197] LustreError: 66587:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2126.807401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2126.954114] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2131.284581] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2131.955825] LustreError: 66588:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2131.975689] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2137.058755] LustreError: 66587:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2140.607833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2140.966190] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2142.012619] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2145.077799] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2154.567992] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2162.064371] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2171.873766] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2171.886687] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2171.914643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2175.699079] Lustre: server umount lustre-MDT0000 complete [ 2178.033544] LustreError: 66588:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2178.065133] LustreError: 66588:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2180.031834] LustreError: 65486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781274898 with bad export cookie 3910754386580719909 [ 2180.044696] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2180.048598] LustreError: 65486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2180.551273] Lustre: server umount lustre-MDT0001 complete [ 2187.584058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2197.919816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2199.456188] Lustre: 63665:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781274901/real 1781274901] req@000000003e4c2f91 x1867801917721216/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781274917 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2199.497575] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2199.517582] Lustre: Skipped 2 previous similar messages [ 2201.568240] Lustre: 63662:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781274903/real 1781274903] req@00000000cb92aafd x1867801917721344/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781274919 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2204.645883] Lustre: 63663:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781274906/real 1781274906] req@00000000a4f70f34 x1867801917721472/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781274922 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2205.665500] LustreError: 63661:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000004fa06702 x1867801917722240/t0(0) o250->MGC192.168.204.159@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2206.098372] LustreError: 69242:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2206.199699] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2206.321327] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2207.264211] Lustre: 63665:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781274908/real 1781274908] req@0000000036d0de96 x1867801917721728/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781274924 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2211.131144] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2211.300028] Lustre: 63664:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781274913/real 1781274913] req@0000000015c6abf5 x1867801917721856/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781274929 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2219.342107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2219.495970] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2219.503993] Lustre: Skipped 4 previous similar messages [ 2224.185319] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2225.139961] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2228.266707] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2231.528534] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2236.703992] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2243.334585] Lustre: server umount lustre-MDT0000 complete [ 2243.556811] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2243.563861] LustreError: 69242:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2243.573507] Lustre: Skipped 2 previous similar messages [ 2243.595561] LustreError: 69242:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2247.033760] LustreError: 64732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781274965 with bad export cookie 3910754386580721288 [ 2247.037761] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2247.047723] LustreError: 64732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2248.680193] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2249.488439] Lustre: server umount lustre-MDT0001 complete [ 2255.638753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2259.034764] Lustre: DEBUG MARKER: files compared the same [ 2261.199810] Lustre: server umount lustre-OST0000 complete [ 2276.666695] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 2279.210993] Key type lgssc unregistered [ 2279.459021] LNet: 74465:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2279.465854] LNetError: 74465:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2279.482758] LNet: Removed LNI 192.168.204.159@tcp [ 2280.144491] Key type .llcrypt unregistered [ 2280.147277] Key type ._llcrypt unregistered [ 2302.052307] Key type ._llcrypt registered [ 2302.054607] Key type .llcrypt registered [ 2302.149312] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 2303.194385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2303.207825] alg: No test for adler32 (adler32-zlib) [ 2304.306934] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 2304.556922] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 2306.200207] Key type lgssc registered [ 2307.439296] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2317.678426] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2317.720144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2319.307886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2323.911851] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug +malloc all [ 2329.576085] LustreError: 75884:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2329.593830] LustreError: 75884:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2334.084636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2334.399897] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2338.917407] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug +malloc all [ 2346.339877] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2352.163371] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2359.658352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2360.013307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2364.149408] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2366.439216] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug +malloc all [ 2374.259697] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2389.985492] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2389.996701] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2390.001117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2390.026794] Lustre: Skipped 1 previous similar message [ 2390.049568] Lustre: Skipped 1 previous similar message [ 2391.810249] Lustre: server umount lustre-OST0000 complete [ 2395.128298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2395.140134] Lustre: Skipped 1 previous similar message [ 2395.148011] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2395.155908] Lustre: Skipped 1 previous similar message [ 2400.228132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2400.234227] Lustre: Skipped 1 previous similar message [ 2401.344167] Lustre: server umount lustre-MDT0000 complete [ 2404.945132] LustreError: 75865:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781275122 with bad export cookie 17071375584431798310 [ 2404.969431] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2405.256904] Lustre: server umount lustre-MDT0001 complete [ 2416.230786] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 2419.026721] Key type lgssc unregistered [ 2419.332164] LNet: 79414:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2419.347429] LNetError: 79414:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2420.394969] LNet: Removed LNI 192.168.204.159@tcp [ 2421.177825] Key type .llcrypt unregistered [ 2421.180713] Key type ._llcrypt unregistered [ 2445.141316] Key type ._llcrypt registered [ 2445.146156] Key type .llcrypt registered [ 2445.281933] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 2446.954574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2446.981171] alg: No test for adler32 (adler32-zlib) [ 2448.258748] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 2448.495650] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 2450.216207] Key type lgssc registered [ 2451.848724] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2460.679089] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2460.693190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2476.384623] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2481.504276] LustreError: 80815:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.159@tcp: failed processing log, type 4: rc = -110 [ 2508.192453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2514.818353] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2522.627032] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2530.969522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2531.098388] Lustre: *** cfs_fail_loc=706, val=0*** [ 2531.497583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2531.553092] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2535.955319] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2536.940494] LustreError: 80841:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2536.987431] LustreError: 80841:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2542.057245] LustreError: 80840:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2544.908326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2545.545990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2551.442591] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2561.461896] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2569.442372] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2572.772220] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2572.779422] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2572.790046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2576.359036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2576.366033] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2576.383106] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2577.888764] Lustre: server umount lustre-OST0000 complete [ 2586.594461] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2586.598295] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2586.614328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2586.614352] Lustre: Skipped 1 previous similar message [ 2586.640791] Lustre: Skipped 1 previous similar message [ 2591.716637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2591.731090] Lustre: Skipped 2 previous similar messages [ 2596.320340] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2596.833803] LustreError: 81906:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2596.871569] LustreError: 81906:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2598.535506] Lustre: server umount lustre-MDT0000 complete [ 2601.966675] LustreError: 81902:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2601.987064] LustreError: 81902:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2602.574367] LustreError: 80821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781275320 with bad export cookie 12559333755935983316 [ 2602.592627] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2602.593984] LustreError: 80821:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2602.974083] Lustre: server umount lustre-MDT0001 complete [ 2615.691200] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 2618.722832] Key type lgssc unregistered [ 2619.083593] LNet: 84432:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2619.092138] LNetError: 84432:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2620.137394] LNet: Removed LNI 192.168.204.159@tcp [ 2621.072217] Key type .llcrypt unregistered [ 2621.074656] Key type ._llcrypt unregistered [ 2647.060167] Key type ._llcrypt registered [ 2647.065235] Key type .llcrypt registered [ 2647.259074] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 2648.805603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2648.834077] alg: No test for adler32 (adler32-zlib) [ 2650.042985] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 2650.338155] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 2652.032650] Key type lgssc registered [ 2654.211923] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2663.743865] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2663.757057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2669.737923] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2680.821181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2681.427682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2686.435118] LustreError: 86540:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2686.463478] LustreError: 86540:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2687.846537] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2691.567763] LustreError: 86543:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2691.609225] LustreError: 86543:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2696.674916] LustreError: 86540:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2696.707633] LustreError: 86540:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2697.841037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2698.195990] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2703.331600] LustreError: 86539:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2703.340534] LustreError: 86539:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2704.607637] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2711.525785] LustreError: 86539:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2711.554367] LustreError: 86539:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2713.668434] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2713.693894] Lustre: MGS: Client 5dca7b9c-bf46-4a29-aed2-b0f4736acb86 (at 0@lo) reconnecting [ 2713.734592] LustreError: 86554:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2714.044626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2714.102046] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2719.182675] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2734.562890] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2734.592858] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2739.234172] Lustre: server umount lustre-OST0000 complete [ 2744.804822] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2744.805304] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2744.829081] Lustre: Skipped 1 previous similar message [ 2744.837058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2744.848309] Lustre: Skipped 2 previous similar messages [ 2749.556850] Lustre: server umount lustre-MDT0000 complete [ 2749.924877] LustreError: 86540:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2749.942155] LustreError: 86540:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2753.984511] Lustre: server umount lustre-MDT0001 complete [ 2758.073357] Lustre: server umount MGS complete [ 2769.429390] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 10:44:45 (1781275485) [ 2799.642597] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 2810.317856] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 2845.928195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2850.860749] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2861.367376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2861.632195] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2861.740571] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2861.897481] Lustre: lustre-MDT0001: new disk, initializing [ 2861.983670] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2862.028598] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2862.050308] Lustre: 93710:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2862.068308] LustreError: 93710:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2866.921803] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2877.401270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2877.704789] Lustre: lustre-OST0000: new disk, initializing [ 2877.714730] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2877.724694] Lustre: 94475:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2877.802992] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2885.154048] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2888.801412] Lustre: 94932:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@000000006d2a3aa9 x1867802600319232/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1781275622 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2894.856327] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2903.139422] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2903.159956] Lustre: MGS: Client 8a209390-e9e9-4c87-8920-b7250c4b088b (at 0@lo) reconnecting [ 2903.241548] Lustre: 95521:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2903.265689] LustreError: 93696:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2903.277478] LustreError: 93696:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2903.303257] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2903.435597] Lustre: lustre-MDT0000: new disk, initializing [ 2903.643782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2903.703908] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2903.744714] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2904.801894] Lustre: 95548:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000000c6cce x1867802600325760/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1781275638 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2909.180956] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2909.199791] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2909.949639] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 2910.109412] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2910.448480] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 2939.878852] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2939.886244] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2939.898251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2939.914714] Lustre: Skipped 2 previous similar messages [ 2942.315175] Lustre: server umount lustre-OST0000 complete [ 2950.116808] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2950.117105] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2950.120967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2950.120976] Lustre: Skipped 1 previous similar message [ 2950.148307] Lustre: Skipped 1 previous similar message [ 2955.234647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2955.248241] Lustre: Skipped 2 previous similar messages [ 2960.352114] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2960.552516] Lustre: server umount lustre-MDT0000 complete [ 2964.456020] LustreError: 93682:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2964.477237] LustreError: 93682:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2971.925747] Lustre: server umount lustre-MDT0001 complete [ 2976.417893] Lustre: server umount MGS complete [ 2989.401109] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 10:48:25 (1781275705) [ 3004.488857] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 3007.576775] Key type lgssc unregistered [ 3008.072077] LNet: 98050:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3008.085441] LNetError: 98050:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3008.109679] LNet: Removed LNI 192.168.204.159@tcp [ 3008.900473] Key type .llcrypt unregistered [ 3008.909168] Key type ._llcrypt unregistered [ 3029.818457] Key type ._llcrypt registered [ 3029.822292] Key type .llcrypt registered [ 3030.004277] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 3031.422211] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3031.471767] alg: No test for adler32 (adler32-zlib) [ 3032.658347] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 3032.972390] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 3034.696180] Key type lgssc registered [ 3036.044187] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3043.852409] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3043.880936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3045.703605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3051.104661] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3055.077167] LustreError: 99274:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3055.100676] LustreError: 99274:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3060.196268] LustreError: 99273:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3060.557489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3060.904548] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3066.492471] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3074.303969] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3080.357596] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3087.887431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3088.402093] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3090.498316] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 3095.208240] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3103.090884] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3116.011472] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3116.017270] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3116.029240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3116.044294] Lustre: Skipped 1 previous similar message [ 3120.601810] Lustre: server umount lustre-OST0000 complete [ 3126.248311] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3126.258381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3126.267057] Lustre: Skipped 1 previous similar message [ 3126.291325] Lustre: Skipped 2 previous similar messages [ 3130.527113] Lustre: server umount lustre-MDT0000 complete [ 3131.369109] LustreError: 99269:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3131.402204] LustreError: 99269:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3134.812321] LustreError: 99253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781275852 with bad export cookie 9317588890964490559 [ 3134.830451] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3134.836350] LustreError: 99253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3135.301168] Lustre: server umount lustre-MDT0001 complete [ 3147.259659] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 3150.159565] Key type lgssc unregistered [ 3150.441299] LNet: 102806:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3150.453568] LNetError: 102806:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3150.478831] LNet: Removed LNI 192.168.204.159@tcp [ 3151.388706] Key type .llcrypt unregistered [ 3151.390567] Key type ._llcrypt unregistered [ 3168.356185] Key type ._llcrypt registered [ 3168.358829] Key type .llcrypt registered [ 3168.478754] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 3169.935628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3169.960182] alg: No test for adler32 (adler32-zlib) [ 3171.088032] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 3171.267912] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 3172.896184] Key type lgssc registered [ 3174.393376] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3180.872439] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3180.876915] LustreError: 103890:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 3180.887584] Lustre: Skipped 1 previous similar message [ 3180.887608] LustreError: 103889:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 3180.908824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3180.914111] LustreError: 103890:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 3180.934910] LustreError: 103890:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3180.952223] LustreError: 103890:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3182.460315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3190.379691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3190.887902] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3196.409338] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3202.761761] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing lsmod [ 3206.091974] LustreError: 105093:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 3206.135783] LustreError: 105098:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 3206.146857] LustreError: 105093:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4955 [ 3206.163965] LustreError: 105093:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3206.170695] LustreError: 105093:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3206.175693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3206.602517] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3207.671430] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 3211.240572] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3211.254463] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3211.282414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3212.776027] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3212.776228] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3212.785246] Lustre: Skipped 1 previous similar message [ 3212.807116] Lustre: Skipped 2 previous similar messages [ 3215.827235] Lustre: server umount lustre-MDT0000 complete [ 3215.847941] LustreError: 103930:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3215.880571] LustreError: 103930:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3219.884281] LustreError: 103905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781275937 with bad export cookie 5197263459429888707 [ 3219.892692] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3219.912214] LustreError: 103905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3220.354836] Lustre: server umount lustre-MDT0001 complete [ 3234.743687] Lustre: server umount lustre-OST0000 complete [ 3243.784444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3244.322333] LustreError: 106163:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3244.392562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3248.745416] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3249.640532] LustreError: 106164:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3254.755609] LustreError: 106163:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3257.871202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3258.298649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3263.504185] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3270.801854] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3276.039843] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3284.500405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3284.798151] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3289.959377] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:97) [ 3291.177973] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3299.063694] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3310.561597] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3310.574310] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3310.595471] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3314.657679] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3314.662417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3314.669457] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3314.694292] Lustre: Skipped 1 previous similar message [ 3316.683898] Lustre: server umount lustre-OST0000 complete [ 3324.897789] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3324.901618] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3324.913828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3324.944735] Lustre: Skipped 1 previous similar message [ 3326.870473] Lustre: server umount lustre-MDT0000 complete [ 3330.017113] LustreError: 106164:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3330.045632] LustreError: 106164:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3330.740352] LustreError: 106145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781276048 with bad export cookie 5197263459429889645 [ 3330.769798] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3331.194457] Lustre: server umount lustre-MDT0001 complete [ 3343.072230] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 3346.183759] Key type lgssc unregistered [ 3346.567930] LNet: 109699:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3346.576460] LNetError: 109699:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3346.593237] LNet: Removed LNI 192.168.204.159@tcp [ 3347.558390] Key type .llcrypt unregistered [ 3347.563295] Key type ._llcrypt unregistered [ 3372.007202] Key type ._llcrypt registered [ 3372.010810] Key type .llcrypt registered [ 3372.099581] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 3373.322165] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3373.373200] alg: No test for adler32 (adler32-zlib) [ 3374.351565] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 3374.548555] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 3376.224220] Key type lgssc registered [ 3377.347018] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3385.634921] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3385.684593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3387.281431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3392.591289] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3397.615757] LustreError: 111118:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3397.655356] LustreError: 111118:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3402.223049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3402.695500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3407.479321] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3415.325493] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3421.636431] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3428.698070] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3429.147324] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3432.250543] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3435.570530] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3442.553339] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3451.885113] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3455.780640] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3463.145582] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3463.161617] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3463.178602] LustreError: Skipped 1 previous similar message [ 3463.180135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3463.231655] Lustre: Skipped 1 previous similar message [ 3465.869227] Lustre: server umount lustre-OST0000 complete [ 3473.389373] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3473.401747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3473.401756] Lustre: Skipped 1 previous similar message [ 3473.413265] Lustre: Skipped 1 previous similar message [ 3476.450723] Lustre: server umount lustre-MDT0000 complete [ 3478.497990] LustreError: 112857:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3478.532622] LustreError: 112857:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3480.172976] LustreError: 113648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781276198 with bad export cookie 6371894193259332701 [ 3480.191455] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3480.444714] Lustre: server umount lustre-MDT0001 complete [ 3491.888836] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 3495.125449] Key type lgssc unregistered [ 3495.498241] LNet: 114755:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3495.506544] LNetError: 114755:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3495.532287] LNet: Removed LNI 192.168.204.159@tcp [ 3496.469299] Key type .llcrypt unregistered [ 3496.471579] Key type ._llcrypt unregistered [ 3513.147740] Key type ._llcrypt registered [ 3513.149633] Key type .llcrypt registered [ 3513.249783] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 3514.158409] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3514.295223] alg: No test for adler32 (adler32-zlib) [ 3515.402970] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 3515.676432] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 3517.352815] Key type lgssc registered [ 3518.547107] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3526.900484] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3526.959707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3528.509911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3533.471713] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3538.915604] LustreError: 115980:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3538.949026] LustreError: 115980:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3542.641375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3543.227936] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3548.460722] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3555.661799] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3561.160763] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3568.516343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3569.003402] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3573.105954] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3575.793826] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3583.404133] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3595.750934] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3595.773178] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3595.788462] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3598.832198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3598.836938] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3598.862996] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3600.923961] Lustre: server umount lustre-OST0000 complete [ 3609.060842] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3609.064647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3609.074397] Lustre: Skipped 1 previous similar message [ 3609.092307] Lustre: Skipped 2 previous similar messages [ 3610.829655] Lustre: server umount lustre-MDT0000 complete [ 3614.187578] LustreError: 115981:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3614.199776] LustreError: 115981:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3614.654573] LustreError: 115962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781276332 with bad export cookie 15502569677473587834 [ 3614.670703] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3615.145283] Lustre: server umount lustre-MDT0001 complete [ 3626.439561] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 3630.213502] Key type lgssc unregistered [ 3630.669934] LNet: 119514:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3630.689373] LNetError: 119514:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3630.736971] LNet: Removed LNI 192.168.204.159@tcp [ 3632.136375] Key type .llcrypt unregistered [ 3632.138305] Key type ._llcrypt unregistered [ 3658.717828] Key type ._llcrypt registered [ 3658.719251] Key type .llcrypt registered [ 3658.823300] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 3659.969710] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3660.194352] alg: No test for adler32 (adler32-zlib) [ 3661.410040] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 3661.655361] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 3663.456247] Key type lgssc registered [ 3665.084186] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3673.593738] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3673.615112] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3675.151580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3679.875317] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3685.346703] LustreError: 120932:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3685.383302] LustreError: 120932:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3689.365436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3689.860159] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3694.273350] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3701.512257] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3706.548158] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3713.711734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3714.035049] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3720.233693] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:163 to 0x240000401:193) [ 3720.391347] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3728.121306] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3735.279778] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3738.809568] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3738.819727] Lustre: Skipped 2 previous similar messages [ 3742.802287] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3742.817623] Lustre: Skipped 2 previous similar messages [ 3745.373670] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3745.432215] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3753.192912] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.59@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3753.202543] Lustre: Skipped 2 previous similar messages [ 3759.671377] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.59@tcp 192.168.0.[2,10]@tcp [ 3763.242386] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.59/24@tcp in log lustre-MDT0000 [ 3763.247765] Lustre: Skipped 2 previous similar messages [ 3767.876512] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.59/24@tcp [ 3767.882948] Lustre: Skipped 1 previous similar message [ 3773.925379] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3773.926990] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3773.944432] LustreError: Skipped 1 previous similar message [ 3773.982558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3776.484868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3776.486441] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3776.502314] LustreError: Skipped 4 previous similar messages [ 3776.510330] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3781.608397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3781.613456] Lustre: Skipped 2 previous similar messages [ 3786.723450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3787.232264] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3787.386647] Lustre: server umount lustre-OST0000 complete [ 3791.843551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3791.854421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3791.857915] Lustre: Skipped 1 previous similar message [ 3791.862290] Lustre: Skipped 1 previous similar message [ 3797.226468] Lustre: server umount lustre-MDT0000 complete [ 3800.932782] LustreError: 120911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781276518 with bad export cookie 6098902263199155090 [ 3800.939536] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3800.945969] LustreError: 120911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3801.256055] Lustre: server umount lustre-MDT0001 complete [ 3812.726861] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 3816.294897] Key type lgssc unregistered [ 3816.766722] LNet: 125782:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3816.781856] LNetError: 125782:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3816.813763] LNet: Removed LNI 192.168.204.159@tcp [ 3817.840853] Key type .llcrypt unregistered [ 3817.843513] Key type ._llcrypt unregistered [ 3846.806314] Key type ._llcrypt registered [ 3846.813794] Key type .llcrypt registered [ 3847.109508] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 3849.090816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3849.248330] alg: No test for adler32 (adler32-zlib) [ 3850.354786] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 3850.529804] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 3852.176210] Key type lgssc registered [ 3853.861071] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3862.638624] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3862.692883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3864.317911] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3864.337694] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.59/24@tcp [ 3864.376877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3870.371425] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3874.788482] LustreError: 127351:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3874.806564] LustreError: 127351:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3879.910418] LustreError: 127352:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3880.140591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3880.676677] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3880.714934] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.59/24@tcp [ 3880.796684] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3884.907514] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3893.265792] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3900.062246] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3906.531334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3907.023600] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3909.132865] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:201 to 0x240000401:225) [ 3913.636828] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 3921.891506] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3934.690159] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3934.699215] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3934.704815] LustreError: Skipped 1 previous similar message [ 3934.732619] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3935.720861] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3935.723380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3935.738454] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3935.750236] Lustre: Skipped 1 previous similar message [ 3940.487513] Lustre: server umount lustre-OST0000 complete [ 3945.954312] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3945.973581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3945.984720] Lustre: Skipped 1 previous similar message [ 3950.790608] Lustre: server umount lustre-MDT0000 complete [ 3951.075034] LustreError: 127351:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3951.087721] LustreError: 127351:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3955.177112] LustreError: 127346:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3955.210526] LustreError: 127346:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3955.314240] LustreError: 127331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781276673 with bad export cookie 15399991979178836924 [ 3955.322845] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3955.332976] LustreError: 127331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3955.634249] Lustre: server umount lustre-MDT0001 complete [ 3969.002286] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 3971.547803] Key type lgssc unregistered [ 3971.745546] LNet: 130943:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3971.756130] LNetError: 130943:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3972.782885] LNet: Removed LNI 192.168.204.159@tcp [ 3973.466589] Key type .llcrypt unregistered [ 3973.472416] Key type ._llcrypt unregistered [ 4004.288824] Key type ._llcrypt registered [ 4004.290871] Key type .llcrypt registered [ 4004.486462] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 4019.026245] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 4019.871762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4019.882661] alg: No test for adler32 (adler32-zlib) [ 4020.818667] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 4020.996410] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 4022.648262] Key type lgssc registered [ 4023.407893] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4057.984225] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4058.008375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4059.461969] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4059.529726] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4059.590332] Lustre: lustre-MDT0000: new disk, initializing [ 4059.678535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4059.700960] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4065.247868] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4076.889770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4076.984589] Lustre: 134343:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4077.035884] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4077.041761] Lustre: Skipped 1 previous similar message [ 4077.136315] Lustre: lustre-MDT0001: new disk, initializing [ 4077.261322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4077.312821] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4077.333670] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4081.292900] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4089.250344] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4094.441917] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4100.475895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4100.701089] Lustre: lustre-OST0000: new disk, initializing [ 4100.703930] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4100.710504] Lustre: 136309:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4100.831925] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4107.497925] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4110.922660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4110.931921] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4111.152288] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4118.173389] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4123.804349] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4124.106130] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4129.839306] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4130.155706] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4136.931183] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4136.932575] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4136.952451] LustreError: Skipped 1 previous similar message [ 4136.989239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4137.013602] Lustre: Skipped 1 previous similar message [ 4138.776375] Lustre: server umount lustre-OST0000 complete [ 4144.111534] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4144.120125] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4144.134267] Lustre: Skipped 1 previous similar message [ 4144.138373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4147.173272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4147.175702] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4149.568948] Lustre: server umount lustre-MDT0000 complete [ 4153.400142] LustreError: 134335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781276871 with bad export cookie 12444159580657040971 [ 4153.414014] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4153.420641] LustreError: 134335:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4153.955130] Lustre: server umount lustre-MDT0001 complete [ 4162.968784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4163.384248] LustreError: 138529:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4163.411944] LustreError: 138529:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4163.489369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4168.697599] LustreError: 138530:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4168.808603] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4173.793859] LustreError: 138529:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4178.614875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4178.945341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4183.881776] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4191.774422] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4197.507211] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4203.581207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4204.025029] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4211.116831] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4219.667916] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4224.807768] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4224.939148] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4229.376177] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4229.470354] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4238.697481] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4238.872766] Lustre: lustre-OST0001: new disk, initializing [ 4238.879633] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4238.885378] Lustre: 141900:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4238.975293] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4243.804661] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4247.141925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4247.151553] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4247.247329] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4251.313544] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4251.509401] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4255.014434] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4255.206948] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4259.795600] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff98ae45d7e000.ost_server_uuid 50 [ 4261.053215] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff98ae45d7e000.ost_server_uuid in FULL state after 0 sec [ 4273.130610] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4273.147915] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4273.149549] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4273.156144] Lustre: Skipped 2 previous similar messages [ 4278.255386] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4278.266033] Lustre: Skipped 1 previous similar message [ 4282.338536] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4282.415494] Lustre: server umount lustre-OST0001 complete [ 4283.373929] LustreError: 141610:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4283.394804] LustreError: 141610:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4288.497110] LustreError: 141610:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4288.501247] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4288.506954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4288.514781] LustreError: 141610:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4288.545950] Lustre: Skipped 2 previous similar messages [ 4298.729568] LustreError: 140350:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4298.769486] LustreError: 140350:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4300.256191] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4300.418586] Lustre: server umount lustre-OST0000 complete [ 4303.842854] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4308.961564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4308.970394] Lustre: Skipped 8 previous similar messages [ 4309.951550] Lustre: server umount lustre-MDT0000 complete [ 4313.325353] LustreError: 138509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781277031 with bad export cookie 12444159580657041853 [ 4313.333914] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4313.335320] LustreError: 138509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4313.715803] Lustre: server umount lustre-MDT0001 complete [ 4331.472186] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 4334.969298] Key type lgssc unregistered [ 4335.248733] LNet: 144906:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4335.260392] LNetError: 144906:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4336.296044] LNet: Removed LNI 192.168.204.159@tcp [ 4337.070991] Key type .llcrypt unregistered [ 4337.073083] Key type ._llcrypt unregistered [ 4377.094838] Key type ._llcrypt registered [ 4377.097220] Key type .llcrypt registered [ 4377.204826] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 4390.441557] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 4391.740307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4391.756485] alg: No test for adler32 (adler32-zlib) [ 4392.792484] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 4392.981961] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 4394.641939] Key type lgssc registered [ 4395.505901] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4426.062408] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4426.088380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4427.400691] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4427.423917] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4427.503666] Lustre: lustre-MDT0000: new disk, initializing [ 4427.585249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4427.610585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4431.065177] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4443.103276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4443.238903] Lustre: 149053:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4443.295390] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4443.298783] Lustre: Skipped 1 previous similar message [ 4443.407971] Lustre: lustre-MDT0001: new disk, initializing [ 4443.523866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4443.558172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4443.568402] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4447.618139] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4456.506645] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4461.582359] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4467.045412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4467.283189] Lustre: lustre-OST0000: new disk, initializing [ 4467.292791] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4467.299821] Lustre: 151027:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4467.373165] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4472.706923] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4472.882728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4472.886577] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4472.961613] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4482.491957] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4487.528880] Lustre: Failing over lustre-OST0000 [ 4487.913765] Lustre: server umount lustre-OST0000 complete [ 4504.544587] Lustre: 147282:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781277206/real 1781277206] req@00000000a8bfddf5 x1867804426928512/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781277222 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4504.544727] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4504.578860] Lustre: 147282:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4504.586150] Lustre: Skipped 1 previous similar message [ 4506.534382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4507.041822] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4507.061706] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4510.050171] Lustre: 147282:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781277211/real 1781277211] req@000000001420b445 x1867804426930048/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781277227 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4510.091179] Lustre: 147282:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4512.678891] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4514.148324] Lustre: 147285:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781277216/real 1781277216] req@0000000033d1528e x1867804426931200/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781277232 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4514.155789] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4514.185333] Lustre: 147285:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4515.012309] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4515.014365] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4516.066341] Lustre: Failing over lustre-MDT0000 [ 4516.358815] Lustre: server umount lustre-MDT0000 complete [ 4519.136434] Lustre: 147284:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781277221/real 1781277221] req@000000004e955b10 x1867804426932224/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781277237 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4519.174020] Lustre: 147284:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4519.400183] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4519.420041] Lustre: Skipped 2 previous similar messages [ 4524.514891] LustreError: 149066:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4524.537760] LustreError: 149066:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 4525.204136] LustreError: 149062:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4529.643549] LustreError: 149061:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4529.677125] LustreError: 149061:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4534.755553] LustreError: 149063:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4534.786045] LustreError: 149063:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4534.933088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4535.016273] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4535.248266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4535.295400] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4535.448555] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4539.655882] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4540.405966] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4540.412685] Lustre: Skipped 1 previous similar message [ 4540.428189] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4540.479802] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4550.629631] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4550.636496] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4550.639348] Lustre: Skipped 1 previous similar message [ 4550.649815] Lustre: Skipped 1 previous similar message [ 4555.749652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4555.756406] Lustre: Skipped 1 previous similar message [ 4560.352347] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4560.473786] Lustre: server umount lustre-OST0000 complete [ 4565.986146] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4566.003670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4566.014050] Lustre: Skipped 1 previous similar message [ 4571.110663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4571.117924] Lustre: Skipped 1 previous similar message [ 4576.228394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4576.236844] Lustre: Skipped 1 previous similar message [ 4578.273040] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4578.640663] Lustre: server umount lustre-MDT0000 complete [ 4580.329888] LustreError: 150218:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4580.357208] LustreError: 150218:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4583.556829] LustreError: 149047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781277301 with bad export cookie 2814771589376908362 [ 4583.559967] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4583.574644] LustreError: 149047:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4584.143354] Lustre: server umount lustre-MDT0001 complete [ 4596.081434] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 4599.099352] Key type lgssc unregistered [ 4599.483707] LNet: 154682:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4599.489933] LNetError: 154682:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4599.521821] LNet: Removed LNI 192.168.204.159@tcp [ 4600.433673] Key type .llcrypt unregistered [ 4600.440408] Key type ._llcrypt unregistered [ 4636.518274] Key type ._llcrypt registered [ 4636.521074] Key type .llcrypt registered [ 4636.645909] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 4637.683507] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4637.818164] alg: No test for adler32 (adler32-zlib) [ 4638.928387] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 4639.169591] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 4640.873432] Key type lgssc registered [ 4642.050766] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4649.824214] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4649.856447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4651.291222] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4655.005245] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4661.735701] LustreError: 156202:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4661.757323] LustreError: 156202:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4662.698623] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4666.666303] LustreError: 156230:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4666.678513] LustreError: 156230:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4666.691849] LustreError: 156230:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 0, failed: rc = -5 [ 4667.051037] Lustre: server umount lustre-MDT0000 complete [ 4681.440606] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 4691.460457] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 4729.600978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4729.871816] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4729.910845] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4730.017288] Lustre: lustre-MDT0000: new disk, initializing [ 4730.083140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4730.130069] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4734.954253] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4745.701161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4745.902769] Lustre: 159534:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4745.991606] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4745.997291] Lustre: Skipped 1 previous similar message [ 4746.093380] Lustre: lustre-MDT0001: new disk, initializing [ 4746.193988] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4746.235874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4746.244974] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4751.019718] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4760.529653] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4766.674379] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4773.132194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4773.327314] Lustre: lustre-OST0000: new disk, initializing [ 4773.330115] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4773.333944] Lustre: 161501:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4773.404689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4779.940564] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 4783.681575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4783.724450] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4783.873619] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4789.683414] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4793.316844] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4794.879650] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 11:18:31 (1781277511) [ 5019.622736] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5019.629158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5019.652443] Lustre: Skipped 2 previous similar messages [ 5019.669249] Lustre: Skipped 2 previous similar messages [ 5024.657755] Lustre: server umount lustre-MDT0000 complete [ 5024.738143] LustreError: 162008:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5024.759167] LustreError: 162008:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5028.180541] LustreError: 159524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781277746 with bad export cookie 11842440797616896259 [ 5028.182591] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5028.185923] LustreError: 159524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5029.864693] LustreError: 159540:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5029.869833] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5029.908096] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5033.889672] LustreError: 159541:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5033.889766] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5034.589604] Lustre: server umount lustre-MDT0001 complete [ 5052.384309] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5052.645429] Lustre: server umount lustre-OST0000 complete [ 5060.466523] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 5070.403236] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 5103.306027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5103.559643] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5103.596405] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5103.704647] Lustre: lustre-MDT0000: new disk, initializing [ 5103.807915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5103.833356] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5108.044083] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5117.575705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5117.695427] Lustre: 166162:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5117.818607] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5117.826851] Lustre: Skipped 1 previous similar message [ 5117.954505] Lustre: lustre-MDT0001: new disk, initializing [ 5118.072343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5118.100883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5118.120864] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5122.507268] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5131.070674] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5135.890751] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5141.924415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5142.170846] Lustre: lustre-OST0000: new disk, initializing [ 5142.173505] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5142.186605] Lustre: 168128:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5142.267188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5147.020428] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5150.275015] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5150.302899] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5150.439493] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5157.611395] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5162.432778] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5162.755644] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5167.118178] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5167.439716] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5171.176038] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5171.181986] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5171.190650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5171.198928] Lustre: Skipped 1 previous similar message [ 5175.870018] Lustre: server umount lustre-OST0000 complete [ 5179.362310] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5179.371166] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5179.395835] Lustre: Skipped 1 previous similar message [ 5179.403365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5191.651090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5191.659949] Lustre: Skipped 5 previous similar messages [ 5193.184361] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5193.377491] Lustre: server umount lustre-MDT0000 complete [ 5196.579894] LustreError: 168127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781277914 with bad export cookie 11842440797617055089 [ 5196.582519] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5196.592545] LustreError: 168127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5196.898973] Lustre: server umount lustre-MDT0001 complete [ 5203.687796] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:25:20 (1781277920) [ 5217.049095] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 5225.862531] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 5257.015156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5257.282751] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5257.342133] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5257.423796] Lustre: lustre-MDT0000: new disk, initializing [ 5257.481451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5257.506136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5261.843869] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5271.719915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5271.847228] Lustre: 172883:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5272.130415] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5276.226136] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5285.539359] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5291.504337] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5297.830702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5298.097346] Lustre: lustre-OST0000: new disk, initializing [ 5298.100777] Lustre: Skipped 1 previous similar message [ 5298.109080] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5298.113285] Lustre: Skipped 2 previous similar messages [ 5298.121465] Lustre: 174860:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5304.109714] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5306.964482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5306.978200] Lustre: Skipped 1 previous similar message [ 5306.983879] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5307.160385] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5312.428578] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5322.731803] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5322.732882] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5322.746045] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5322.751561] Lustre: Skipped 2 previous similar messages [ 5322.754917] LustreError: Skipped 1 previous similar message [ 5324.902036] Lustre: server umount lustre-OST0000 complete [ 5328.353360] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5332.966728] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5332.980571] Lustre: Skipped 1 previous similar message [ 5334.296114] Lustre: server umount lustre-MDT0000 complete [ 5337.125203] LustreError: 172873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781278055 with bad export cookie 11842440797617055971 [ 5337.150607] LustreError: 172873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5337.160201] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5337.383583] Lustre: server umount lustre-MDT0001 complete [ 5343.952984] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:27:40 (1781278060) [ 5355.698327] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 5363.324778] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 5393.772801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5393.961405] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5393.984810] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5394.045943] Lustre: lustre-MDT0000: new disk, initializing [ 5394.123271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5394.130244] Lustre: Skipped 2 previous similar messages [ 5394.153771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5397.723668] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5406.318607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5406.389054] Lustre: 179177:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5406.597741] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5410.096372] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5418.158259] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5423.224689] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5429.118703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5429.411810] Lustre: 181155:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5434.325749] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5436.534694] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5442.042659] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5449.474427] Lustre: server umount lustre-OST0000 complete [ 5452.776474] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5452.784347] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5452.798986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5452.810624] Lustre: Skipped 4 previous similar messages [ 5463.120975] Lustre: server umount lustre-MDT0000 complete [ 5466.283795] LustreError: 179172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781278184 with bad export cookie 11842440797617057056 [ 5466.285619] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5466.294265] LustreError: 179172:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5477.027518] Lustre: server umount lustre-MDT0001 complete [ 5489.070704] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 5491.936554] Key type lgssc unregistered [ 5492.301851] LNet: 183013:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5492.313610] LNetError: 183013:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5492.338601] LNet: Removed LNI 192.168.204.159@tcp [ 5493.508384] Key type .llcrypt unregistered [ 5493.510043] Key type ._llcrypt unregistered [ 5516.897352] Key type ._llcrypt registered [ 5516.900962] Key type .llcrypt registered [ 5516.994904] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 5517.847042] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5517.871523] alg: No test for adler32 (adler32-zlib) [ 5518.921455] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 5519.194769] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 5520.920187] Key type lgssc registered [ 5521.882965] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5528.578536] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5528.594227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5530.048227] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5534.166417] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5540.326735] LustreError: 184431:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5540.346794] LustreError: 184431:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5541.053541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5541.354487] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5544.401667] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5550.581852] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5555.586635] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5561.175666] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5561.521227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5567.053632] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5574.126042] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5575.862688] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5596.641286] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5596.655961] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5596.656635] Lustre: Skipped 1 previous similar message [ 5596.670308] Lustre: Skipped 1 previous similar message [ 5600.948882] Lustre: server umount lustre-OST0000 complete [ 5606.887753] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5606.902853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5606.911456] Lustre: Skipped 1 previous similar message [ 5610.729356] Lustre: server umount lustre-MDT0000 complete [ 5612.007625] LustreError: 185159:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5612.040070] LustreError: 185159:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5614.089850] LustreError: 185166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781278332 with bad export cookie 3863304946889273739 [ 5614.090725] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5614.099394] LustreError: 185166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5614.421878] Lustre: server umount lustre-MDT0001 complete [ 5623.633364] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 5626.304200] Key type lgssc unregistered [ 5626.551779] LNet: 187985:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5626.564496] LNetError: 187985:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5626.579924] LNet: Removed LNI 192.168.204.159@tcp [ 5627.405715] Key type .llcrypt unregistered [ 5627.411133] Key type ._llcrypt unregistered [ 5649.317434] Key type ._llcrypt registered [ 5649.321460] Key type .llcrypt registered [ 5649.442335] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 5650.150462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5650.182593] alg: No test for adler32 (adler32-zlib) [ 5651.143776] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 5651.444058] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 5653.152190] Key type lgssc registered [ 5653.909908] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5660.062641] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5660.088350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5661.567616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5665.308354] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5671.907758] LustreError: 189410:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5671.922955] LustreError: 189410:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5672.645481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5673.035736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5677.429766] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5683.484149] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5688.371728] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5694.038867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5694.425791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5699.010796] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5706.314672] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5706.760310] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5712.868448] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5712.883656] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5712.904084] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5713.593595] Lustre: lustre-OST0000: Not available for connect from 192.168.204.59@tcp (stopping) [ 5716.970108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5716.973879] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5716.993937] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5718.243937] Lustre: server umount lustre-OST0000 complete [ 5725.557434] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff98ae49fab800.ost_server_uuid 50 [ 5726.933379] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5742.561926] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5742.580133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5742.586414] Lustre: Skipped 2 previous similar messages [ 5746.897067] Lustre: server umount lustre-MDT0000 complete [ 5748.713313] LustreError: 189410:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5748.726019] LustreError: 189410:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5750.152079] LustreError: 189390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781278468 with bad export cookie 8339174631143481606 [ 5750.155641] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5750.160824] LustreError: 189390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5750.402810] Lustre: server umount lustre-MDT0001 complete [ 5757.297429] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 11:34:33 (1781278473) [ 5762.759542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5763.181648] LustreError: 193320:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5763.241335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5767.143563] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5768.684261] LustreError: 193321:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5773.803560] LustreError: 193320:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5774.337317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5774.727107] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5779.235289] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5785.988655] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5791.571595] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5798.047087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5798.367517] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5803.984226] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5808.624191] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5809.965627] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5816.090706] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5816.315589] Lustre: lustre-OST0001: new disk, initializing [ 5816.319585] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5816.325144] Lustre: 196160:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5816.476125] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5824.640917] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5826.772537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5826.785159] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5826.993931] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5837.868560] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5842.410082] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5842.417691] Lustre: Skipped 1 previous similar message [ 5842.423375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5842.435100] Lustre: Skipped 3 previous similar messages [ 5846.689324] Lustre: server umount lustre-OST0000 complete [ 5847.534239] LustreError: 195140:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5847.562578] LustreError: 195140:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5852.211254] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5852.495638] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5857.581907] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5857.899728] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5864.092450] LustreError: 195140:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5864.127057] LustreError: 195140:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 5873.123163] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5873.125569] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5873.127558] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5873.137714] Lustre: Skipped 2 previous similar messages [ 5873.164082] Lustre: Skipped 2 previous similar messages [ 5877.941253] Lustre: server umount lustre-OST0001 complete [ 5882.339436] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5882.352213] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5893.601447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5893.613800] Lustre: Skipped 6 previous similar messages [ 5896.160305] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5896.397530] Lustre: server umount lustre-MDT0000 complete [ 5898.725217] LustreError: 193317:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5898.740685] LustreError: 193317:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5899.444785] LustreError: 196158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781278617 with bad export cookie 8339174631143482677 [ 5899.447407] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5899.454418] LustreError: 196158:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5899.859641] Lustre: server umount lustre-MDT0001 complete [ 5922.335129] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 11:37:18 (1781278638) [ 5929.288630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5929.682507] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5929.735361] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5929.926254] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5934.761418] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5942.981335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5943.056397] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5943.085334] Lustre: 199598:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5947.730208] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5955.704993] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5961.408093] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5967.608588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5967.752820] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5973.674930] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5974.381598] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 5981.799180] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5988.080265] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5988.205601] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5994.580544] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6002.138053] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6006.762864] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6006.773607] Lustre: Skipped 1 previous similar message [ 6006.777529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6006.791569] Lustre: Skipped 1 previous similar message [ 6011.101341] Lustre: server umount lustre-OST0000 complete [ 6011.885726] LustreError: 203154:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6011.911568] LustreError: 203154:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6026.721517] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6032.312608] Lustre: server umount lustre-OST0001 complete [ 6050.272989] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6050.476783] Lustre: server umount lustre-MDT0000 complete [ 6053.707589] LustreError: 201468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781278771 with bad export cookie 8339174631143484084 [ 6053.710940] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6053.715602] LustreError: 201468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6054.010258] Lustre: server umount lustre-MDT0001 complete [ 6073.855472] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 11:39:49 (1781278789) [ 6087.476867] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 6097.094770] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 6131.182192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6131.636433] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6131.659344] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6131.755813] Lustre: lustre-MDT0000: new disk, initializing [ 6131.905168] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6131.909233] Lustre: Skipped 3 previous similar messages [ 6131.928170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6135.554770] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6145.390829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6145.551485] Lustre: 207697:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6145.620445] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6145.628927] Lustre: Skipped 1 previous similar message [ 6145.698699] Lustre: lustre-MDT0001: new disk, initializing [ 6145.867165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6145.884214] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6151.075920] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6160.668396] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6166.292791] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6173.309303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6173.644595] Lustre: lustre-OST0000: new disk, initializing [ 6173.649525] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6173.658308] Lustre: 209666:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6180.678610] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6183.507578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6183.526626] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6183.632227] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6189.689367] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6194.220514] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6194.483697] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6199.523321] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6199.798584] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6204.388501] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6204.391790] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6204.394324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6204.394331] Lustre: Skipped 10 previous similar messages [ 6204.410237] Lustre: Skipped 6 previous similar messages [ 6204.415296] LustreError: Skipped 1 previous similar message [ 6216.160863] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6216.412756] Lustre: server umount lustre-OST0000 complete [ 6234.082367] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6234.352132] Lustre: server umount lustre-MDT0000 complete [ 6235.114173] LustreError: 211004:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6235.168732] LustreError: 211004:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 6238.792408] LustreError: 208491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781278956 with bad export cookie 8339174631143485645 [ 6238.793265] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6238.812079] LustreError: 208491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6239.101950] Lustre: server umount lustre-MDT0001 complete [ 6248.735293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6252.928937] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6262.670822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6263.461949] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6263.477191] Lustre: Skipped 3 previous similar messages [ 6268.071666] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6274.821265] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6280.385935] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6287.173613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6293.474574] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6301.122778] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6305.201765] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6305.387328] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6309.987129] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6310.224530] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6315.489116] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6315.492531] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6315.529255] Lustre: Skipped 3 previous similar messages [ 6326.240097] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6326.429054] Lustre: server umount lustre-OST0000 complete [ 6332.725969] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6336.248293] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6367.212233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6372.476630] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6378.904471] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6380.844338] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6380.863920] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6380.865446] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6383.841785] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 6391.269460] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6391.274116] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6391.277296] Lustre: Skipped 17 previous similar messages [ 6394.140893] Lustre: server umount lustre-OST0000 complete [ 6404.325236] Lustre: server umount lustre-MDT0000 complete [ 6407.344046] LustreError: 211863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781279125 with bad export cookie 8339174631143486527 [ 6407.346963] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6407.352888] LustreError: 211863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6407.727556] Lustre: server umount lustre-MDT0001 complete [ 6417.207630] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 11:45:33 (1781279133) [ 6424.984073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6429.649262] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6436.757954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6441.297281] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6447.784675] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6451.863988] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6457.351510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6462.937257] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6469.562720] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6474.562413] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6474.838871] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6479.383614] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6479.697066] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6485.591196] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6485.753687] Lustre: lustre-OST0001: new disk, initializing [ 6485.759280] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6485.764574] Lustre: 221220:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6491.754471] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6492.237587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6492.257679] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6492.396892] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6500.672346] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6505.154701] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6505.387760] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6509.806931] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6510.089258] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6513.127221] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6513.146959] LustreError: Skipped 2 previous similar messages [ 6513.149076] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6513.163522] Lustre: Skipped 4 previous similar messages [ 6517.990686] Lustre: server umount lustre-OST0001 complete [ 6518.243994] LustreError: 219674:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6518.266243] LustreError: 219674:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 6524.029375] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6524.217214] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6528.804813] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6529.119501] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6556.567584] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6556.897983] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6556.906110] Lustre: Skipped 6 previous similar messages [ 6561.222827] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6562.278729] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6562.298755] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6562.299256] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6562.322770] Lustre: Skipped 1 previous similar message [ 6566.936157] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6568.156075] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6584.289174] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6584.432140] Lustre: server umount lustre-OST0001 complete [ 6607.403941] Lustre: server umount lustre-OST0000 complete [ 6617.362304] Lustre: server umount lustre-MDT0000 complete [ 6621.002137] LustreError: 217834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781279339 with bad export cookie 8339174631143487486 [ 6621.014664] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6621.017374] LustreError: 217834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6627.378126] Lustre: server umount lustre-MDT0001 complete [ 6650.795815] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 11:49:27 (1781279367) [ 6656.399755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6656.634501] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6656.662150] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6661.285792] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6669.088296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6669.190761] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6669.249821] Lustre: 226461:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6674.643630] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6682.125389] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6687.880852] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6694.285678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6694.631519] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6700.413911] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6706.073675] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6713.340697] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6713.427652] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6718.857197] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6728.579966] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6730.123820] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6735.563983] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6735.828565] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6740.662567] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6740.954184] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6745.749446] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff98ae504f1800.ost_server_uuid 50 [ 6747.060818] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff98ae504f1800.ost_server_uuid in IDLE state after 0 sec [ 6747.951560] Lustre: Permanently deactivating lustre-OST0001 [ 6747.973906] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6751.135312] Lustre: Permanently reactivating lustre-OST0001 [ 6754.786439] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6754.787977] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6754.791319] LustreError: Skipped 6 previous similar messages [ 6754.794517] Lustre: Skipped 20 previous similar messages [ 6761.014319] Lustre: server umount lustre-OST0001 complete [ 6772.172396] Lustre: server umount lustre-OST0000 complete [ 6780.441280] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6780.455288] Lustre: Skipped 11 previous similar messages [ 6785.079389] Lustre: server umount lustre-MDT0000 complete [ 6790.207285] LustreError: 226453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781279508 with bad export cookie 8339174631143488781 [ 6790.211659] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6790.217397] LustreError: 226453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6790.571602] Lustre: server umount lustre-MDT0001 complete [ 6819.260947] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 11:52:15 (1781279535) [ 6826.917076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6827.204902] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6827.239869] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6827.246602] Lustre: Skipped 5 previous similar messages [ 6831.219673] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6837.956552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6838.042319] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6838.070301] Lustre: 233157:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6841.762178] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6847.303989] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6852.330362] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6858.302296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6858.421981] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6858.460822] Lustre: Permanently deactivating lustre-OST0000 [ 6858.470927] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6865.548635] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6868.038136] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6868.125454] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6868.126069] LustreError: 235507:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -5 [ 6872.328675] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6878.625101] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6878.739552] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6884.184346] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 6886.760806] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6894.157964] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6896.727011] Lustre: Permanently reactivating lustre-OST0000 [ 6896.755694] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6896.762795] Lustre: Skipped 2 previous similar messages [ 6905.964687] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6905.972992] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6906.011824] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6906.088115] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6906.124789] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6921.424394] Lustre: server umount lustre-OST0001 complete [ 6931.746528] Lustre: server umount lustre-OST0000 complete [ 6945.012820] Lustre: server umount lustre-MDT0000 complete [ 6949.418369] LustreError: 233148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781279667 with bad export cookie 8339174631143490426 [ 6949.433143] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6949.441104] LustreError: 233148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6950.224738] Lustre: server umount lustre-MDT0001 complete [ 6963.221926] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 6966.516927] Key type lgssc unregistered [ 6966.935824] LNet: 238037:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6966.944328] LNetError: 238037:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6966.982625] LNet: Removed LNI 192.168.204.159@tcp [ 6967.941678] Key type .llcrypt unregistered [ 6967.943742] Key type ._llcrypt unregistered [ 6992.567501] Key type ._llcrypt registered [ 6992.571420] Key type .llcrypt registered [ 6992.660995] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 6993.781941] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6993.842032] alg: No test for adler32 (adler32-zlib) [ 6995.088737] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 6995.541106] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 6997.336241] Key type lgssc registered [ 6998.897863] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7010.477344] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7010.504115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7012.108761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7017.076732] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7022.566407] LustreError: 239506:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7022.588146] LustreError: 239506:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7026.905596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7027.033432] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7027.096844] Lustre: 239494:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7027.105290] Lustre: Permanently deactivating lustre-MDT0001 [ 7027.148450] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7027.503666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7028.301931] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7028.312262] LustreError: 239537:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7028.335518] LustreError: 239537:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7028.357090] LustreError: 239537:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 1, failed: rc = -108 [ 7032.662834] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7040.402706] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7046.891697] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7053.484744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7053.867476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7057.982077] LustreError: 241350:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7058.005431] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7059.984975] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7063.017975] LustreError: 241348:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7067.193888] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7068.137123] LustreError: 241612:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7072.234345] LustreError: 241612:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7072.256774] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7072.257608] LustreError: 241612:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7074.124873] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7074.394494] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7079.415793] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7079.424975] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7080.610981] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7087.252853] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7089.807531] Lustre: Permanently reactivating lustre-MDT0001 [ 7089.819345] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7089.828223] Lustre: Skipped 2 previous similar messages [ 7095.984875] Lustre: lustre-MDT0001: Client d4567a2a-c5e4-4297-bfaf-1581a626ea6b (at 192.168.204.59@tcp) reconnecting [ 7101.731263] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7103.376869] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7109.123624] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7109.295972] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7112.992358] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7113.181155] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7115.660514] Lustre: Permanently deactivating lustre-MDT0001 [ 7115.677983] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7115.684215] Lustre: Skipped 2 previous similar messages [ 7124.574378] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7134.693104] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7134.698259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7134.724682] Lustre: Skipped 2 previous similar messages [ 7134.737840] Lustre: Skipped 1 previous similar message [ 7139.826109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7139.846054] Lustre: Skipped 3 previous similar messages [ 7140.004451] Lustre: server umount lustre-MDT0000 complete [ 7143.177924] LustreError: 239487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781279861 with bad export cookie 9751923051263924854 [ 7143.181042] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7143.190280] LustreError: 239487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7143.425632] Lustre: server umount lustre-MDT0001 complete [ 7149.231235] Lustre: server umount lustre-OST0000 complete [ 7162.723915] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 11:57:58 (1781279878) [ 7176.616493] Lustre: server umount lustre-OST0001 complete [ 7183.399534] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 7193.689689] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 7226.407247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7226.705813] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7226.716093] Lustre: Skipped 2 previous similar messages [ 7226.776856] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7226.888523] Lustre: lustre-MDT0000: new disk, initializing [ 7227.003509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7227.031907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7231.646052] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7241.504650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7241.614238] Lustre: 247810:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7241.665540] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7241.669529] Lustre: Skipped 1 previous similar message [ 7241.734269] Lustre: lustre-MDT0001: new disk, initializing [ 7241.839465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7241.881252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7241.911526] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7245.496628] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7253.115827] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7257.418201] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7263.356388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7263.560862] Lustre: lustre-OST0000: new disk, initializing [ 7263.563329] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7263.566905] Lustre: 249776:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7263.650311] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7270.104830] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7273.525683] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7273.537728] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7273.611363] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7279.516651] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7281.448850] LustreError: 248629:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7288.313508] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7288.442771] Lustre: lustre-OST0001: new disk, initializing [ 7288.449385] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7288.458090] Lustre: 250890:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7291.458157] LustreError: 248629:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7291.490038] LustreError: 247817:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7294.375245] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7297.084556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7297.092519] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7297.224365] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7301.474734] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7301.513354] LustreError: 247817:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7302.715221] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7305.899088] Lustre: lustre-OST0001: Not available for connect from 192.168.204.59@tcp (stopping) [ 7307.748152] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7307.748656] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7307.754305] LustreError: Skipped 1 previous similar message [ 7307.770859] Lustre: Skipped 2 previous similar messages [ 7310.263368] Lustre: server umount lustre-OST0001 complete [ 7311.010640] LustreError: 250278:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7328.230587] LustreError: 249758:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7328.237734] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7328.241280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7328.241289] Lustre: Skipped 2 previous similar messages [ 7328.260101] LustreError: 249758:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 7328.331357] Lustre: Skipped 1 previous similar message [ 7332.112813] Lustre: server umount lustre-OST0000 complete [ 7338.472699] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7338.491814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7338.503280] Lustre: Skipped 2 previous similar messages [ 7341.812752] Lustre: server umount lustre-MDT0000 complete [ 7345.104646] LustreError: 247802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781280063 with bad export cookie 9751923051263927668 [ 7345.108393] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7345.112684] LustreError: 247802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7351.347301] Lustre: server umount lustre-MDT0001 complete [ 7359.603202] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 7362.369744] Key type lgssc unregistered [ 7362.691640] LNet: 252942:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7362.697811] LNetError: 252942:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7362.719138] LNet: Removed LNI 192.168.204.159@tcp [ 7363.508746] Key type .llcrypt unregistered [ 7363.515356] Key type ._llcrypt unregistered [ 7397.203090] Key type ._llcrypt registered [ 7397.216813] Key type .llcrypt registered [ 7397.338090] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 7398.416854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7398.522939] alg: No test for adler32 (adler32-zlib) [ 7399.560192] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 7399.758939] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 7401.472289] Key type lgssc registered [ 7402.443919] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7409.609171] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7409.642602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7411.004428] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7411.027265] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7411.195675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7415.000932] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7422.438382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7422.614687] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7422.652114] Lustre: 255147:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7422.919975] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7426.941569] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7433.471730] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7438.448857] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7445.069682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7445.279808] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7445.453180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7451.272430] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7453.336124] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7458.312229] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7509.474806] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7509.479511] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7509.493380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7509.989347] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7510.015246] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7510.021456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7515.000923] Lustre: server umount lustre-OST0000 complete [ 7519.533490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7532.840467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7533.128935] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7540.138777] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7547.943829] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7550.509335] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7550.523608] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7550.532438] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7591.401949] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7591.406191] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7591.423959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7593.448660] Lustre: server umount lustre-OST0000 complete [ 7597.025263] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7597.036676] LustreError: Skipped 1 previous similar message [ 7597.043352] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7597.063679] Lustre: Skipped 1 previous similar message [ 7597.068451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7597.071114] Lustre: Skipped 1 previous similar message [ 7601.633504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7601.633559] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7610.850097] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7610.955969] Lustre: server umount lustre-MDT0000 complete [ 7613.775609] LustreError: 255138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781280331 with bad export cookie 12516333464991824184 [ 7613.776439] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7613.787289] LustreError: 255138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7614.017654] Lustre: server umount lustre-MDT0001 complete [ 7625.790509] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 7630.476991] Key type lgssc unregistered [ 7631.308645] LNet: 260570:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7631.342921] LNetError: 260570:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7631.376624] LNet: Removed LNI 192.168.204.159@tcp [ 7632.873981] Key type .llcrypt unregistered [ 7632.881293] Key type ._llcrypt unregistered [ 7662.415946] Key type ._llcrypt registered [ 7662.417587] Key type .llcrypt registered [ 7662.511258] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 7663.302748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7663.322775] alg: No test for adler32 (adler32-zlib) [ 7664.342765] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 7664.573181] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 7666.176183] Key type lgssc registered [ 7667.504606] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7677.127374] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7677.157439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7679.002605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7685.458432] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7688.173148] LustreError: 261993:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7688.191295] LustreError: 261993:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7693.302359] LustreError: 261992:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7695.662314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7696.069267] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7700.624257] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7708.249508] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7713.199373] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7720.656909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7720.999233] LustreError: 263813:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7721.249716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7723.339559] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7727.790629] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7735.119468] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7764.452093] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7764.454559] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7764.459904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7764.468813] Lustre: Skipped 1 previous similar message [ 7764.490535] LustreError: Skipped 1 previous similar message [ 7766.308462] Lustre: server umount lustre-OST0000 complete [ 7769.572109] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7769.576757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7769.587224] Lustre: Skipped 1 previous similar message [ 7769.600086] Lustre: Skipped 2 previous similar messages [ 7774.689289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7774.693140] Lustre: Skipped 1 previous similar message [ 7775.650179] Lustre: server umount lustre-MDT0000 complete [ 7779.081289] LustreError: 261974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781280497 with bad export cookie 8747049327712678261 [ 7779.088518] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7779.094310] LustreError: 261974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7779.293580] Lustre: server umount lustre-MDT0001 complete [ 7790.586127] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 7794.040901] Key type lgssc unregistered [ 7794.401700] LNet: 266303:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7794.411636] LNetError: 266303:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7794.437153] LNet: Removed LNI 192.168.204.159@tcp [ 7795.502416] Key type .llcrypt unregistered [ 7795.509279] Key type ._llcrypt unregistered [ 7814.550760] Key type ._llcrypt registered [ 7814.556695] Key type .llcrypt registered [ 7814.667574] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 7815.747352] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7815.826199] alg: No test for adler32 (adler32-zlib) [ 7817.019740] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 7817.312980] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 7818.976204] Key type lgssc registered [ 7819.923556] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7828.506990] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7828.518418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7830.054774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7835.465260] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7840.241235] LustreError: 267525:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7840.280708] LustreError: 267525:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7845.349977] LustreError: 267524:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7847.451407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7848.064559] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7853.803821] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7862.919654] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7868.797798] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7876.915278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7877.610669] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7884.970034] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 7893.452157] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7900.149847] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7918.016398] Lustre: server umount lustre-OST0000 complete [ 7924.707076] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7924.715663] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7924.733196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7925.729600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7925.742111] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7930.850856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7930.856346] Lustre: Skipped 1 previous similar message [ 7932.031375] Lustre: server umount lustre-MDT0000 complete [ 7934.947154] Lustre: 266835:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781280636/real 1781280636] req@00000000e7a5b140 x1867808018301952/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1781280652 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7934.977710] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7934.998587] LustreError: 267524:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7935.027409] LustreError: 267524:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7936.850237] LustreError: 270474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781280654 with bad export cookie 8728831504846908152 [ 7936.868066] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7936.871406] LustreError: 270474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7937.467549] Lustre: server umount lustre-MDT0001 complete [ 7953.927602] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 7958.960182] Key type lgssc unregistered [ 7959.492841] LNet: 271247:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7959.499206] LNetError: 271247:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7959.529156] LNet: Removed LNI 192.168.204.159@tcp [ 7960.866533] Key type .llcrypt unregistered [ 7960.877459] Key type ._llcrypt unregistered [ 7990.855079] Key type ._llcrypt registered [ 7990.860189] Key type .llcrypt registered [ 7991.009637] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 7993.078545] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7993.110973] alg: No test for adler32 (adler32-zlib) [ 7994.268892] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 7994.654206] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 7996.504291] Key type lgssc registered [ 7997.999321] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8007.512411] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8007.547295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8009.153980] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8015.132739] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 8019.459295] LustreError: 272666:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8019.487689] LustreError: 272666:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8024.548657] LustreError: 272665:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8024.832531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8025.455799] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8030.859450] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 8039.078260] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8044.735208] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8051.528456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8051.991464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8054.097161] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8059.050484] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 8067.419658] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8095.204681] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8095.214944] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8095.247668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8096.227307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8096.237726] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8096.243168] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8100.730354] Lustre: server umount lustre-OST0000 complete [ 8106.467413] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8106.504235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8106.511622] Lustre: Skipped 2 previous similar messages [ 8110.681025] Lustre: server umount lustre-MDT0000 complete [ 8111.586992] LustreError: 272665:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8111.608510] LustreError: 272665:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8114.691739] LustreError: 272645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781280832 with bad export cookie 15487926180544211799 [ 8114.718039] LustreError: 272645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8114.718202] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8115.088074] Lustre: server umount lustre-MDT0001 complete [ 8129.317187] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 8132.855350] Key type lgssc unregistered [ 8133.105722] LNet: 276972:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8133.116670] LNetError: 276972:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8133.145732] LNet: Removed LNI 192.168.204.159@tcp [ 8134.363529] Key type .llcrypt unregistered [ 8134.377105] Key type ._llcrypt unregistered [ 8154.869180] Key type ._llcrypt registered [ 8154.871835] Key type .llcrypt registered [ 8154.984494] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 8156.229640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8156.264958] alg: No test for adler32 (adler32-zlib) [ 8157.331422] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 8157.592279] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 8159.264224] Key type lgssc registered [ 8160.530133] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8168.728137] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8168.759371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8170.421847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8175.903146] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 8180.721657] LustreError: 278210:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8180.746883] LustreError: 278210:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8185.413210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8185.746653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8190.968132] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 8199.202187] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8204.831407] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8211.568187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8212.042668] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8215.159554] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8218.181199] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 8225.940788] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8239.080816] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 8239.089898] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8239.108315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8241.123489] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8241.127120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8241.136890] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8245.282455] Lustre: server umount lustre-OST0000 complete [ 8251.404409] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8251.413912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8251.425293] Lustre: Skipped 1 previous similar message [ 8251.450453] Lustre: Skipped 2 previous similar messages [ 8256.484542] LustreError: 278191:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8256.512205] LustreError: 278191:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8256.788800] Lustre: server umount lustre-MDT0000 complete [ 8260.971633] LustreError: 278175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781280978 with bad export cookie 3460936569693045997 [ 8260.979837] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8260.987745] LustreError: 278175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8261.430526] Lustre: server umount lustre-MDT0001 complete [ 8271.843519] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 8274.544575] Key type lgssc unregistered [ 8274.905732] LNet: 281913:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8274.917120] LNetError: 281913:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8274.941276] LNet: Removed LNI 192.168.204.159@tcp [ 8275.795311] Key type .llcrypt unregistered [ 8275.797474] Key type ._llcrypt unregistered [ 8286.527227] Key type ._llcrypt registered [ 8286.531197] Key type .llcrypt registered [ 8286.678820] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8304.223969] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 8318.734382] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 8319.832169] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8319.885973] alg: No test for adler32 (adler32-zlib) [ 8320.898479] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 8321.034968] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 8322.691169] Key type lgssc registered [ 8323.939800] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8357.942643] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8357.981655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8359.292282] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8359.319952] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8359.407960] Lustre: lustre-MDT0000: new disk, initializing [ 8359.492293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8359.514611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8363.923997] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 8373.265954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8373.377373] Lustre: 285468:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8373.446874] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8373.457435] Lustre: Skipped 1 previous similar message [ 8373.505698] Lustre: lustre-MDT0001: new disk, initializing [ 8373.581868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8373.611054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8373.617961] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8378.712914] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 8389.568373] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8397.640840] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8405.281288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8405.674971] Lustre: lustre-OST0000: new disk, initializing [ 8405.682365] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8405.698578] Lustre: 287434:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8405.801251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8412.578592] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 8413.274312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8413.289924] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8413.424678] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8422.184819] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8426.993770] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8427.290642] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8431.847614] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8432.116947] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8434.153846] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8434.154117] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8434.160657] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8434.172306] Lustre: Skipped 1 previous similar message [ 8439.267390] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8439.281828] Lustre: Skipped 2 previous similar messages [ 8440.300575] Lustre: server umount lustre-OST0000 complete [ 8444.388302] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8444.393150] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8444.403433] Lustre: Skipped 1 previous similar message [ 8444.415291] Lustre: Skipped 1 previous similar message [ 8449.508185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8449.526241] Lustre: Skipped 1 previous similar message [ 8450.020701] Lustre: server umount lustre-MDT0000 complete [ 8453.383661] LustreError: 287433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781281171 with bad export cookie 17695084236916755165 [ 8453.387923] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8453.389822] LustreError: 287433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8453.657810] Lustre: server umount lustre-MDT0001 complete [ 8462.328767] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 12:19:38 (1781281178) [ 8469.443652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8470.070800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8475.106974] LustreError: 289845:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8475.138613] LustreError: 289845:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8475.208978] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 8480.231150] LustreError: 289846:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8485.349852] LustreError: 289845:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8485.405847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8485.945172] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8491.428763] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 8500.025956] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8506.647638] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8514.417312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8514.735367] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8521.282432] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 8529.869373] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8540.643922] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8540.648310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8540.672099] Lustre: Skipped 1 previous similar message [ 8545.124971] Lustre: server umount lustre-OST0000 complete [ 8550.901101] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8550.913767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8550.924458] Lustre: Skipped 1 previous similar message [ 8550.941364] Lustre: Skipped 2 previous similar messages [ 8563.168173] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8563.325751] Lustre: server umount lustre-MDT0000 complete [ 8565.223534] LustreError: 290789:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8565.257031] LustreError: 290789:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8567.156149] LustreError: 289825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781281285 with bad export cookie 17695084236916756047 [ 8567.179549] LustreError: 289825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8567.180256] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8567.506373] Lustre: server umount lustre-MDT0001 complete [ 8579.111276] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 8582.402352] Key type lgssc unregistered [ 8582.731595] LNet: 293383:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8582.738253] LNetError: 293383:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8582.771174] LNet: Removed LNI 192.168.204.159@tcp [ 8583.877103] Key type .llcrypt unregistered [ 8583.884618] Key type ._llcrypt unregistered [ 8613.074788] Key type ._llcrypt registered [ 8613.082241] Key type .llcrypt registered [ 8613.190492] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 8629.034941] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 8630.203202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8630.431989] alg: No test for adler32 (adler32-zlib) [ 8631.570558] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 8631.819674] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 8633.592180] Key type lgssc registered [ 8634.879566] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8677.827861] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8677.860711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8679.090558] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8679.116663] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8679.162212] Lustre: lustre-MDT0000: new disk, initializing [ 8679.222927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8679.235503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8682.725814] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 8693.717168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8693.874862] Lustre: 297210:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8693.958203] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8693.965586] Lustre: Skipped 1 previous similar message [ 8694.096305] Lustre: lustre-MDT0001: new disk, initializing [ 8694.219048] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8694.256666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8694.276438] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8698.403816] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 8707.690800] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8713.219696] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8719.397697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8719.637464] Lustre: lustre-OST2710: new disk, initializing [ 8719.646793] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8719.652251] Lustre: 299177:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8719.725113] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8725.769541] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 8728.127259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8728.136642] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8728.226330] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8733.892450] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 8739.699672] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8739.846177] Lustre: lustre-OST03e8: new disk, initializing [ 8739.855812] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8739.868365] Lustre: 300242:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8739.949543] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8745.965404] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 8749.688250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8749.695326] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8749.828937] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8753.226347] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 8759.995262] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8760.208300] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8765.525512] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8765.731285] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8770.676482] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8770.882774] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8775.009411] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8775.210358] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8780.773522] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8780.778396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8780.801027] Lustre: Skipped 3 previous similar messages [ 8780.815749] Lustre: Skipped 2 previous similar messages [ 8781.813887] Lustre: server umount lustre-MDT0000 complete [ 8786.313753] LustreError: 299176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781281504 with bad export cookie 10095470030285943391 [ 8786.320507] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8786.327069] LustreError: 299176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8787.117081] Lustre: server umount lustre-MDT0001 complete [ 8801.793749] Lustre: server umount lustre-OST2710 complete [ 8815.128805] Lustre: server umount lustre-OST03e8 complete [ 8832.336652] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 8842.121195] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 8874.732802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8875.078968] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8875.110235] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8875.232843] Lustre: lustre-MDT0000: new disk, initializing [ 8875.385830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8875.428267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8879.775798] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 8889.998664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8890.180705] Lustre: 305489:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8890.261700] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8890.265293] Lustre: Skipped 1 previous similar message [ 8890.336132] Lustre: lustre-MDT0001: new disk, initializing [ 8890.499465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8890.591947] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8890.631540] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8895.849116] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 8905.550868] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8910.978941] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8917.337329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8917.623297] Lustre: lustre-OST0000: new disk, initializing [ 8917.625625] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8917.630309] Lustre: 307457:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8917.710603] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8922.812294] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 8925.763598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8925.771690] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8925.867443] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8932.912639] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8937.578528] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8937.809187] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8942.868175] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8943.135329] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8946.660714] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8946.666289] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8946.677331] Lustre: Skipped 1 previous similar message [ 8946.706451] Lustre: Skipped 1 previous similar message [ 8951.388366] Lustre: server umount lustre-OST0000 complete [ 8956.898788] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8956.900157] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8956.905174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8956.916133] Lustre: Skipped 1 previous similar message [ 8956.923561] Lustre: Skipped 1 previous similar message [ 8961.729129] Lustre: server umount lustre-MDT0000 complete [ 8962.022991] LustreError: 305502:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8962.046668] LustreError: 305502:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 8965.487962] LustreError: 305481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781281683 with bad export cookie 10095470030285945071 [ 8965.499823] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8965.789897] Lustre: server umount lustre-MDT0001 complete [ 8973.298153] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 12:28:09 (1781281689) [ 8974.740811] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8976.293276] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 12:28:12 (1781281692) [ 8979.425634] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 9000.090611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9000.312954] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9000.331100] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9000.482049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9004.583224] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9012.450448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9012.547748] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9012.576400] Lustre: 311312:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9016.642534] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9037.618095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9037.795589] Lustre: Denying initial registration attempt from nid 192.168.204.159@tcp, specified as failover [ 9037.803973] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9037.812799] LustreError: 312767:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -99 [ 9037.818805] LustreError: 312767:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 9037.823564] LustreError: 312767:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9037.931127] Lustre: server umount lustre-OST0000 complete [ 9037.934445] LustreError: 312767:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9043.427897] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9043.431869] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9043.438931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9050.596586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9050.604855] Lustre: Skipped 2 previous similar messages [ 9057.248167] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9057.375972] Lustre: server umount lustre-MDT0000 complete [ 9060.533627] LustreError: 311306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781281778 with bad export cookie 10095470030285945953 [ 9060.544354] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9060.550672] LustreError: 311306:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9060.826883] Lustre: server umount lustre-MDT0001 complete [ 9070.294639] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 9073.211514] Key type lgssc unregistered [ 9073.640541] LNet: 313889:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9073.654826] LNetError: 313889:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9073.689129] LNet: Removed LNI 192.168.204.159@tcp [ 9075.330311] Key type .llcrypt unregistered [ 9075.332368] Key type ._llcrypt unregistered [ 9089.879609] Key type ._llcrypt registered [ 9089.886338] Key type .llcrypt registered [ 9090.049690] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 9091.123030] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9091.159707] alg: No test for adler32 (adler32-zlib) [ 9092.288869] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 9092.646800] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 9094.360168] Key type lgssc registered [ 9095.703566] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9140.461287] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 9150.251502] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9150.270442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9151.790171] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9151.845466] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9152.161224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9156.400901] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9163.639342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9163.795669] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9163.842657] Lustre: 316670:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9164.056685] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9167.952222] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9174.363969] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9179.472803] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9198.263966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9198.485500] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9198.668259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9204.345684] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9211.081371] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9214.436460] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9214.442839] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9214.458992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9214.952558] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9214.960909] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9214.987330] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9219.294633] Lustre: server umount lustre-OST0000 complete [ 9225.194463] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9225.196398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9225.207263] Lustre: Skipped 1 previous similar message [ 9229.555198] Lustre: server umount lustre-MDT0000 complete [ 9233.431406] LustreError: 316661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781281951 with bad export cookie 661396795590700934 [ 9233.440268] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9233.445500] LustreError: 316661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9233.896573] Lustre: server umount lustre-MDT0001 complete [ 9244.448933] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 9247.039552] Key type lgssc unregistered [ 9247.362764] LNet: 320242:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9247.376405] LNetError: 320242:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9247.393089] LNet: Removed LNI 192.168.204.159@tcp [ 9248.143317] Key type .llcrypt unregistered [ 9248.147805] Key type ._llcrypt unregistered [ 9273.805908] Key type ._llcrypt registered [ 9273.808926] Key type .llcrypt registered [ 9273.965165] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 9275.403634] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9275.446127] alg: No test for adler32 (adler32-zlib) [ 9276.503424] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 9276.724864] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 9278.456250] Key type lgssc registered [ 9279.562391] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9286.931389] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9286.953906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9288.432283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9292.188343] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9298.916281] LustreError: 321661:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9298.940105] LustreError: 321661:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9299.085745] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9299.507311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9304.290392] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9311.393487] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9316.318669] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9324.723656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9325.102524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9331.411878] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9339.681520] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9354.721685] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9354.736263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9354.750594] Lustre: Skipped 2 previous similar messages [ 9354.764307] Lustre: Skipped 2 previous similar messages [ 9358.002770] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.59@tcp (stopping) [ 9359.845570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9359.861589] Lustre: Skipped 2 previous similar messages [ 9360.403790] Lustre: server umount lustre-MDT0000 complete [ 9363.098989] LustreError: 321654:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9363.826075] LustreError: 321641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781282081 with bad export cookie 11603948677201643879 [ 9363.844589] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9363.847602] LustreError: 321641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9364.961826] LustreError: 321655:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9364.964253] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9365.011670] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9368.213241] LustreError: 321684:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9373.095772] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9373.095834] LustreError: 321657:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9373.103424] Lustre: Skipped 2 previous similar messages [ 9373.120796] LustreError: 321657:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9378.272149] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9378.549770] Lustre: server umount lustre-MDT0001 complete [ 9384.949856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9392.985294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9408.998080] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xa10981d77077e6ef [ 9409.009476] Lustre: MGC192.168.204.159@tcp: Connection restored to 0@lo (at 0@lo) [ 9409.351628] LustreError: 323493:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9409.366352] LustreError: 323493:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9409.424515] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9409.497392] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9414.086281] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9422.279778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9422.580332] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9426.604884] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9433.092600] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9433.396654] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9435.213333] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9440.635148] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9442.233953] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9448.422504] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9448.434758] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9448.436879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9448.442953] Lustre: Skipped 2 previous similar messages [ 9452.468927] Lustre: server umount lustre-OST0000 complete [ 9458.658056] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9458.664860] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9458.680761] Lustre: Skipped 2 previous similar messages [ 9462.388934] Lustre: server umount lustre-MDT0000 complete [ 9463.787052] LustreError: 325281:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9463.810802] LustreError: 325281:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9465.912695] LustreError: 324465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781282183 with bad export cookie 11603948677201651439 [ 9465.921554] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9465.926192] LustreError: 324465:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9466.333370] Lustre: server umount lustre-MDT0001 complete [ 9477.700207] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 9480.485733] Key type lgssc unregistered [ 9480.780206] LNet: 327950:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9480.791905] LNetError: 327950:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9481.837208] LNet: Removed LNI 192.168.204.159@tcp [ 9482.784151] Key type .llcrypt unregistered [ 9482.788784] Key type ._llcrypt unregistered [ 9505.434244] Key type ._llcrypt registered [ 9505.437699] Key type .llcrypt registered [ 9505.518524] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 9506.434782] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9506.476255] alg: No test for adler32 (adler32-zlib) [ 9507.465617] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 9507.669943] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 9509.360224] Key type lgssc registered [ 9510.346022] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9517.183796] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9517.204790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9518.715805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9522.989995] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9528.291911] LustreError: 329370:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9528.326753] LustreError: 329370:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9531.692141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9532.147257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9537.037559] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9544.204414] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9550.561683] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9553.891510] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9553.892487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9553.907953] Lustre: Skipped 1 previous similar message [ 9553.922666] Lustre: Skipped 1 previous similar message [ 9557.987383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9557.995916] Lustre: Skipped 1 previous similar message [ 9563.108640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9563.117028] Lustre: Skipped 1 previous similar message [ 9568.225454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9568.225615] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [ 9568.239185] Lustre: Skipped 1 previous similar message [ 9568.525080] Lustre: server umount lustre-MDT0000 complete [ 9572.075106] LustreError: 329348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781282290 with bad export cookie 8394865132818646386 [ 9572.086028] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9572.094994] LustreError: 329348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9572.381930] Lustre: server umount lustre-MDT0001 complete [ 9586.220823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9586.646351] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9586.694993] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9586.916196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9592.394343] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9601.546901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9601.619404] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9601.647958] Lustre: 331956:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9601.915371] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9606.874055] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9614.469847] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9620.272291] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9630.120212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9630.309828] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9630.323929] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9630.334459] LustreError: 333843:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 9630.344285] LustreError: 333843:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 9630.350833] LustreError: 333843:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9630.408386] Lustre: server umount lustre-OST0000 complete [ 9630.415384] LustreError: 333843:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9641.138183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9641.395585] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9641.560425] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9648.136674] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9648.231866] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9654.769720] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9661.831876] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9661.952677] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9662.083208] Lustre: lustre-OST0001: new disk, initializing [ 9662.089474] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9662.106364] Lustre: 335386:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9662.251304] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9668.247261] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9671.823285] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9671.844679] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9671.971726] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9675.373862] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9682.407490] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9682.415979] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9682.428835] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9682.445241] Lustre: Skipped 1 previous similar message [ 9682.466890] Lustre: Skipped 1 previous similar message [ 9684.380854] Lustre: server umount lustre-OST0001 complete [ 9687.545417] LustreError: 334681:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9687.559152] LustreError: 334681:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9689.057322] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9689.068452] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9692.645963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9692.647050] LustreError: 334681:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9692.658094] Lustre: Skipped 3 previous similar messages [ 9692.692222] LustreError: 334681:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9693.775162] Lustre: server umount lustre-OST0000 complete [ 9697.763703] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9697.788040] Lustre: Skipped 2 previous similar messages [ 9711.072267] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9711.379561] Lustre: server umount lustre-MDT0000 complete [ 9713.122721] LustreError: 331968:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9715.122909] LustreError: 331950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781282433 with bad export cookie 8394865132818646939 [ 9715.130769] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9715.144966] LustreError: 331950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9715.666994] Lustre: server umount lustre-MDT0001 complete [ 9725.770040] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [ 9728.878341] Key type lgssc unregistered [ 9729.281627] LNet: 337342:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9729.299105] LNetError: 337342:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9729.332966] LNet: Removed LNI 192.168.204.159@tcp [ 9730.333091] Key type .llcrypt unregistered [ 9730.337300] Key type ._llcrypt unregistered [ 9797.076690] Key type ._llcrypt registered [ 9797.078649] Key type .llcrypt registered [ 9797.172457] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [ 9812.348669] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [ 9813.324341] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9813.506558] alg: No test for adler32 (adler32-zlib) [ 9814.611694] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [ 9814.922537] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [ 9816.640203] Key type lgssc registered [ 9817.840996] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9852.602465] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9852.632454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9853.938704] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9853.985426] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9854.081925] Lustre: lustre-MDT0000: new disk, initializing [ 9854.168846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9854.210451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9858.155102] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9866.881278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9867.009790] Lustre: 342276:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9867.054714] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9867.058720] Lustre: Skipped 1 previous similar message [ 9867.121539] Lustre: lustre-MDT0001: new disk, initializing [ 9867.187378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9867.242303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9867.249389] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9871.044828] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9879.447248] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9884.770222] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9891.043783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9891.350912] Lustre: lustre-OST0000: new disk, initializing [ 9891.360138] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9891.369771] Lustre: 344245:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9891.504970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9897.680268] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9898.609619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9898.617478] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9898.795800] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9906.705786] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9911.334904] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9911.497248] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9915.282507] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9915.459995] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9919.458326] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9919.500684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9919.520619] Lustre: Skipped 1 previous similar message [ 9923.599468] Lustre: server umount lustre-OST0000 complete [ 9928.679716] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9928.686739] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9928.697332] Lustre: Skipped 1 previous similar message [ 9928.707190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9929.701434] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9929.724501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9929.734812] Lustre: Skipped 1 previous similar message [ 9933.984788] Lustre: server umount lustre-MDT0000 complete [ 9937.578453] LustreError: 342268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781282655 with bad export cookie 3588943110467594516 [ 9937.580118] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9937.586281] LustreError: 342268:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9937.980720] Lustre: server umount lustre-MDT0001 complete [ 9946.373197] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 12:44:22 (1781282662) [ 9953.399892] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 12:44:29 (1781282669) [ 9960.458430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9961.042180] LustreError: 347001:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9961.078643] LustreError: 347001:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9961.185320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9965.568697] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9966.561670] LustreError: 347002:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9971.682202] LustreError: 347001:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9972.883053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9973.272819] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9977.400631] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [ 9984.637225] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9989.890299] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9995.955517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9996.334609] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10003.049600] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [10010.225432] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10018.037088] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10019.562924] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10021.258472] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10022.883450] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10024.575405] Lustre: Failing over lustre-MDT0000 [10024.964606] Lustre: server umount lustre-MDT0000 complete [10026.978861] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10026.991617] LustreError: 347001:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10026.996043] Lustre: Skipped 2 previous similar messages [10027.015662] LustreError: 347001:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10032.097644] LustreError: 347002:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10032.120366] LustreError: 347002:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10042.339403] Lustre: 340499:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781282744/real 1781282744] req@00000000a7b57f02 x1867810112342272/t0(0) o400->MGC192.168.204.159@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1781282760 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10042.341326] LustreError: 347001:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10042.357797] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10042.371849] LustreError: 347001:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [10042.619625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10052.584606] LustreError: 340498:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000007f22c940 x1867810112347136/t0(0) o250->MGC192.168.204.159@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10052.843526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10052.900889] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10053.288658] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10056.213104] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [10058.220231] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10058.255840] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [10058.299242] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10063.837816] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10065.209479] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10068.399725] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10069.934100] Lustre: Failing over lustre-MDT0000 [10070.255715] Lustre: server umount lustre-MDT0000 complete [10073.259379] LustreError: 346998:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10073.267710] LustreError: 346998:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [10073.571221] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10073.575340] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10073.597297] Lustre: Skipped 1 previous similar message [10087.523066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10087.615164] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10087.814901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10087.872585] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10088.597396] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10091.885890] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [10093.062759] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10093.067339] Lustre: Skipped 2 previous similar messages [10093.099657] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [10093.157928] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10100.902544] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10102.471400] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10108.385429] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10108.391367] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10108.398568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10108.412825] Lustre: Skipped 2 previous similar messages [10113.507497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10113.524111] Lustre: Skipped 4 previous similar messages [10114.109081] Lustre: server umount lustre-MDT0000 complete [10117.665972] LustreError: 346983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781282835 with bad export cookie 3588943110467597225 [10117.669465] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10117.683721] LustreError: 346983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10118.281721] Lustre: server umount lustre-MDT0001 complete [10131.162275] Lustre: server umount lustre-OST0000 complete [10146.539115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10146.898899] LustreError: 353339:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10146.913352] LustreError: 353339:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [10150.016377] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [10156.313136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10156.569857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10156.574026] Lustre: Skipped 1 previous similar message [10159.927276] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [10165.421394] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10169.646694] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10175.122281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10180.390894] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [10186.852864] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10191.857836] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10196.961601] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10196.966410] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10196.975625] LustreError: Skipped 1 previous similar message [10196.981312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10196.999928] Lustre: Skipped 2 previous similar messages [10198.189373] Lustre: server umount lustre-OST0000 complete [10207.922692] Lustre: server umount lustre-MDT0000 complete [10211.306899] LustreError: 353319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781282929 with bad export cookie 3588943110467597974 [10211.311290] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10211.323130] LustreError: 353319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10211.656793] Lustre: server umount lustre-MDT0001 complete [10221.673332] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [10224.173561] Key type lgssc unregistered [10224.494861] LNet: 356869:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10224.504896] LNetError: 356869:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10224.524044] LNet: Removed LNI 192.168.204.159@tcp [10225.317790] Key type .llcrypt unregistered [10225.319900] Key type ._llcrypt unregistered [10249.480846] Key type ._llcrypt registered [10249.482898] Key type .llcrypt registered [10249.585135] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [10250.297346] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10250.333325] alg: No test for adler32 (adler32-zlib) [10251.400635] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [10251.621436] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [10253.256282] Key type lgssc registered [10254.652408] Lustre: Echo OBD driver; http://www.lustre.org/ [10261.913246] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10261.938694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10263.354256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10266.597429] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [10273.515488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10273.764653] LustreError: 358388:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10273.797452] LustreError: 358388:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10274.074028] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10278.327607] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [10284.650405] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10289.723582] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10295.850133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10296.258667] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10301.852771] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [10308.013516] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10308.550504] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10318.820975] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10318.825041] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10318.827693] Lustre: Skipped 2 previous similar messages [10318.842064] Lustre: Skipped 2 previous similar messages [10323.220271] Lustre: server umount lustre-MDT0000 complete [10323.939771] LustreError: 358387:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10323.951598] LustreError: 358387:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10326.513054] LustreError: 360965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781283044 with bad export cookie 1330736190389255946 [10326.518563] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10326.521082] LustreError: 360965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10326.968818] Lustre: server umount lustre-MDT0001 complete [10340.231883] Lustre: server umount lustre-OST0000 complete [10353.353916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10353.822698] LustreError: 362095:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10353.884763] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10358.184923] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [10359.271174] LustreError: 362096:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10363.394461] LustreError: 362095:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10367.008256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10367.365082] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10371.323620] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [10378.687806] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10382.881151] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10388.462260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10388.745883] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10394.495441] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [10398.961969] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10400.900702] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10413.538666] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10413.547902] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10413.558494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10414.564296] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10414.565874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10414.577864] Lustre: Skipped 1 previous similar message [10414.588450] Lustre: Skipped 1 previous similar message [10417.009283] Lustre: server umount lustre-MDT0000 complete [10419.682459] LustreError: 363338:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10419.692535] LustreError: 363338:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10420.217404] LustreError: 362077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781283138 with bad export cookie 1330736190389257059 [10420.226526] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10420.537468] Lustre: server umount lustre-MDT0001 complete [10434.080529] Lustre: server umount lustre-OST0000 complete [10448.259705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10448.768890] LustreError: 366008:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10448.872710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10453.678548] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [10461.549277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10465.916792] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [10472.394817] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10478.607845] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10486.010528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10486.312480] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10486.316753] Lustre: Skipped 1 previous similar message [10492.861922] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [10493.523885] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10499.308234] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10503.649310] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10503.656356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10503.661295] Lustre: Skipped 1 previous similar message [10503.678389] Lustre: Skipped 1 previous similar message [10508.779065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10509.131039] Lustre: server umount lustre-OST0000 complete [10513.377614] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10513.384793] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10518.574639] Lustre: server umount lustre-MDT0000 complete [10519.009223] LustreError: 366736:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10519.032636] LustreError: 366736:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10521.788926] LustreError: 366744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781283239 with bad export cookie 1330736190389258081 [10521.792969] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10521.805247] LustreError: 366744:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [10522.095657] Lustre: server umount lustre-MDT0001 complete [10530.598477] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [10532.717457] Key type lgssc unregistered [10532.929550] LNet: 369540:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10532.940619] LNetError: 369540:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10532.962466] LNet: Removed LNI 192.168.204.159@tcp [10533.616535] Key type .llcrypt unregistered [10533.618906] Key type ._llcrypt unregistered [10553.367934] Key type ._llcrypt registered [10553.372571] Key type .llcrypt registered [10553.474959] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [10554.103505] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10554.133905] alg: No test for adler32 (adler32-zlib) [10555.154629] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [10555.340609] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [10557.008176] Key type lgssc registered [10557.888605] Lustre: Echo OBD driver; http://www.lustre.org/ [10564.481525] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10564.508065] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10564.518748] LustreError: 370978:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10564.543668] LustreError: 370978:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10564.550890] LustreError: 370978:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10564.556442] LustreError: 370978:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10564.562541] LustreError: 370978:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [10571.832319] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10571.846217] LustreError: 371375:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10571.876791] LustreError: 371375:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10571.886101] LustreError: 371375:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10571.893282] LustreError: 371375:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10571.905594] LustreError: 371375:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [10589.996738] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [10592.776593] Key type lgssc unregistered [10593.117093] LNet: 372378:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10593.124458] LNetError: 372378:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10593.147380] LNet: Removed LNI 192.168.204.159@tcp [10593.964736] Key type .llcrypt unregistered [10593.973360] Key type ._llcrypt unregistered [10610.303685] Key type ._llcrypt registered [10610.308302] Key type .llcrypt registered [10610.426291] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [10626.142248] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [10627.104040] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10627.160788] alg: No test for adler32 (adler32-zlib) [10628.285349] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [10628.591852] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [10630.312266] Key type lgssc registered [10631.332281] Lustre: Echo OBD driver; http://www.lustre.org/ [10667.365666] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10667.376901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10668.720528] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10668.775276] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10668.927223] Lustre: lustre-MDT0000: new disk, initializing [10669.047638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10669.072503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10672.917429] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [10682.713760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10682.828529] Lustre: 375536:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10682.890037] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10682.895723] Lustre: Skipped 1 previous similar message [10682.996924] Lustre: lustre-MDT0001: new disk, initializing [10683.108251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10683.161853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10683.185783] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10687.456613] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [10695.801802] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10700.557452] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10706.939723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10707.136039] Lustre: lustre-OST0000: new disk, initializing [10707.139051] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10707.146212] Lustre: 377504:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10707.260498] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10712.841964] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [10714.150109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10714.162636] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10714.289647] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10721.611452] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10726.462640] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10726.696371] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10731.197398] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10731.496945] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10734.562774] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10734.570827] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10734.593432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10735.097637] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10735.100995] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10735.121414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10735.134951] Lustre: Skipped 1 previous similar message [10739.498317] Lustre: server umount lustre-OST0000 complete [10744.801954] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10744.811946] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10744.828820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10750.437448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10750.445669] Lustre: Skipped 3 previous similar messages [10755.568043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10755.582819] Lustre: Skipped 1 previous similar message [10757.088694] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10757.335486] Lustre: server umount lustre-MDT0000 complete [10761.345498] LustreError: 376341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781283479 with bad export cookie 3300349832548071673 [10761.370301] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10761.375025] LustreError: 376341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10761.992342] Lustre: server umount lustre-MDT0001 complete [10772.342327] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10773.938137] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 12:58:10 (1781283490) [10779.631122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10780.106230] LustreError: 379966:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10780.125945] LustreError: 379966:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10780.180904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10784.252850] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [10785.252873] LustreError: 379967:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10790.369858] LustreError: 379966:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10792.061498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10792.489925] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10797.580067] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [10806.084192] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10811.801648] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10818.601765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10819.006807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10824.310945] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [10830.327656] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10837.271027] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10837.412296] Lustre: lustre-OST0001: new disk, initializing [10837.419796] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10837.437516] Lustre: 382802:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10837.585823] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10844.305982] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [10847.832556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10847.852975] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10848.039776] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10853.896781] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [10858.467128] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10858.469339] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10858.478846] LustreError: Skipped 1 previous similar message [10858.484386] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10858.498167] Lustre: Skipped 2 previous similar messages [10858.523500] Lustre: Skipped 1 previous similar message [10863.063410] Lustre: server umount lustre-OST0001 complete [10863.608328] LustreError: 381786:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10863.647393] LustreError: 381786:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10868.722571] LustreError: 383611:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10868.758339] LustreError: 383611:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10878.946291] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10878.958248] LustreError: 383611:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10878.963162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10878.963172] Lustre: Skipped 1 previous similar message [10878.971736] Lustre: Skipped 1 previous similar message [10878.999182] LustreError: 383611:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10884.682231] Lustre: server umount lustre-OST0000 complete [10889.186894] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10894.187333] Lustre: server umount lustre-MDT0000 complete [10897.668899] LustreError: 379947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781283615 with bad export cookie 3300349832548072555 [10897.678194] LustreError: 379947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10897.689334] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10904.130340] Lustre: server umount lustre-MDT0001 complete [10913.585285] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [10916.133353] Key type lgssc unregistered [10916.460824] LNet: 384792:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10916.469771] LNetError: 384792:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10916.484510] LNet: Removed LNI 192.168.204.159@tcp [10917.424826] Key type .llcrypt unregistered [10917.433330] Key type ._llcrypt unregistered [10968.272568] Key type ._llcrypt registered [10968.274378] Key type .llcrypt registered [10968.353434] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [10969.733456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10969.790209] alg: No test for adler32 (adler32-zlib) [10970.888921] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [10971.163866] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [10972.816550] Key type lgssc registered [10973.704357] Lustre: Echo OBD driver; http://www.lustre.org/ [10980.779030] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10980.819442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10982.095281] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10982.110533] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10982.294239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10986.398068] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [10995.197384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10995.341540] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10995.372392] Lustre: 387604:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10995.663627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11000.513704] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [11007.618417] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11012.828925] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11016.166608] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11016.177298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11016.188299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11017.701037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11017.701142] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11021.643328] Lustre: server umount lustre-MDT0000 complete [11026.655715] LustreError: 387596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781283744 with bad export cookie 209925066217168286 [11026.673134] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11027.528305] Lustre: server umount lustre-MDT0001 complete [11040.138219] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 13:02:35 (1781283755) [11046.059473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11046.421509] LustreError: 390118:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11046.429790] LustreError: 390118:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11046.486903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11051.426232] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [11051.494758] LustreError: 390119:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11056.612815] LustreError: 390118:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11059.371021] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11060.002991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11065.167771] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [11073.705513] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11080.298611] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11088.687305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11088.994344] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11089.421756] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11098.720156] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [11105.669458] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11113.168182] Lustre: Permanently deactivating lustre-OST0000 [11113.183792] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11122.234838] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11124.099383] LustreError: 393152:0:(mgs_llog.c:2041:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11124.105355] LustreError: 393152:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [11124.109195] LustreError: 393152:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11126.736448] LustreError: 393200:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [11126.741814] LustreError: 393200:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11134.429331] Lustre: server umount lustre-OST0000 complete [11137.109601] LustreError: 393449:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [11137.119713] LustreError: 393449:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11139.050420] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11139.073082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11139.075958] Lustre: Skipped 1 previous similar message [11144.161724] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11144.170612] Lustre: Skipped 1 previous similar message [11149.281483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11149.291592] Lustre: Skipped 1 previous similar message [11153.376427] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11153.625509] Lustre: server umount lustre-MDT0000 complete [11154.403227] LustreError: 390844:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11154.425386] LustreError: 390844:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11157.057642] LustreError: 390100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781283875 with bad export cookie 209925066217168867 [11157.060862] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11157.065383] LustreError: 390100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11167.385389] Lustre: server umount lustre-MDT0001 complete [11175.961969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11179.997564] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [11182.579645] LustreError: 394610:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11188.176181] Lustre: 394951:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.159@tcp [11190.244331] Lustre: server umount MGS complete [11203.622057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11204.254715] LustreError: 395675:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11204.293575] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11204.303583] Lustre: Skipped 1 previous similar message [11204.381713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11209.477526] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [11214.819804] LustreError: 395676:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11214.845197] LustreError: 395676:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11217.420816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11217.717572] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11222.151887] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [11228.750636] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11233.979981] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11235.601995] Lustre: Permanently reactivating lustre-OST0000 [11235.618827] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11235.628358] Lustre: Skipped 2 previous similar messages [11263.456192] LustreError: 397455:0:(osp_dev.c:693:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11270.953729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11271.453886] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11271.457554] Lustre: Skipped 1 previous similar message [11278.666426] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [11288.857191] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11294.178829] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11294.187016] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11294.200232] Lustre: Skipped 1 previous similar message [11294.202233] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11294.226427] Lustre: Skipped 1 previous similar message [11295.974768] Lustre: server umount lustre-OST0000 complete [11299.813781] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11299.832380] LustreError: Skipped 1 previous similar message [11299.840416] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11299.868847] Lustre: Skipped 1 previous similar message [11314.146584] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11314.374938] Lustre: server umount lustre-MDT0000 complete [11314.659690] LustreError: 395670:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11314.681909] LustreError: 395670:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11319.686419] LustreError: 397249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781284037 with bad export cookie 209925066217169910 [11319.691621] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11319.705367] LustreError: 397249:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11320.338877] Lustre: server umount lustre-MDT0001 complete [11333.682898] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [11338.277682] Key type lgssc unregistered [11338.704721] LNet: 400091:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11338.719309] LNetError: 400091:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11338.733919] LNet: Removed LNI 192.168.204.159@tcp [11339.966988] Key type .llcrypt unregistered [11339.970532] Key type ._llcrypt unregistered [11360.326461] Key type ._llcrypt registered [11360.328702] Key type .llcrypt registered [11360.438290] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [11377.670470] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [11378.810931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11378.864853] alg: No test for adler32 (adler32-zlib) [11379.946247] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [11380.160555] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [11381.848265] Key type lgssc registered [11383.088820] Lustre: Echo OBD driver; http://www.lustre.org/ [11416.914516] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11416.949335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11418.207218] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11418.234252] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11418.316976] Lustre: lustre-MDT0000: new disk, initializing [11418.413922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11418.433645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11423.157376] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [11434.979511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11435.308477] Lustre: 403246:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11435.441254] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11435.449460] Lustre: Skipped 1 previous similar message [11435.535123] Lustre: lustre-MDT0001: new disk, initializing [11435.744269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11435.858191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11435.884594] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11442.426462] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [11453.622135] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11461.067620] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11468.116133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11468.459186] Lustre: lustre-OST0000: new disk, initializing [11468.462531] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11468.469378] Lustre: 405219:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11468.594686] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11475.553076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11475.562656] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11475.689260] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11476.411805] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [11486.154959] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11491.528614] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11492.032540] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11496.939628] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11497.214148] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11501.551330] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11501.552199] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11501.562844] LustreError: Skipped 1 previous similar message [11501.564987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11501.577822] Lustre: Skipped 1 previous similar message [11505.936872] Lustre: server umount lustre-OST0000 complete [11511.778588] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11511.817338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11511.824342] Lustre: Skipped 1 previous similar message [11516.909621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11516.926075] Lustre: Skipped 1 previous similar message [11522.028451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11525.088141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11525.315038] Lustre: server umount lustre-MDT0000 complete [11528.570696] LustreError: 403240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781284246 with bad export cookie 14077636568351629991 [11528.571725] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11528.590424] LustreError: 403240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11528.833612] Lustre: server umount lustre-MDT0001 complete [11537.223902] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11538.794542] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 13:10:54 (1781284254) [11544.609855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11545.026399] LustreError: 407681:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11545.052043] LustreError: 407681:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11545.147882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11549.463293] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [11550.179168] LustreError: 407682:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11555.307470] LustreError: 407681:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11557.190280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11557.717259] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11562.967671] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [11570.029240] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11574.941910] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11580.675947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11580.975478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11587.063951] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [11594.491895] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11596.585100] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11599.203278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11605.487545] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11605.489053] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11605.501276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11605.514566] Lustre: Skipped 2 previous similar messages [11605.532585] Lustre: Skipped 2 previous similar messages [11610.160661] Lustre: server umount lustre-OST0000 complete [11614.179580] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11614.190178] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11614.208038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11620.153246] Lustre: server umount lustre-MDT0000 complete [11620.835470] LustreError: 407682:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11620.866140] LustreError: 407682:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11623.387658] LustreError: 407664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781284341 with bad export cookie 14077636568351630873 [11623.399149] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11623.722947] Lustre: server umount lustre-MDT0001 complete [11633.949710] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [11636.750875] Key type lgssc unregistered [11637.041924] LNet: 411408:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11637.046924] LNetError: 411408:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11638.119376] LNet: Removed LNI 192.168.204.159@tcp [11638.879713] Key type .llcrypt unregistered [11638.881821] Key type ._llcrypt unregistered [11659.033907] Key type ._llcrypt registered [11659.044755] Key type .llcrypt registered [11659.160698] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [11659.453460] Key type .llcrypt unregistered [11659.458039] Key type ._llcrypt unregistered [11675.466599] Key type ._llcrypt registered [11675.468563] Key type .llcrypt registered [11675.570901] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [11677.043167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11677.076674] alg: No test for adler32 (adler32-zlib) [11678.175399] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [11678.497832] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [11680.188237] Key type lgssc registered [11681.662144] Lustre: Echo OBD driver; http://www.lustre.org/ [11689.308639] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11689.327557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11690.827959] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11695.601132] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [11700.199584] LustreError: 413622:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11700.214162] LustreError: 413622:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11703.146685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11703.552325] LustreError: 413623:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11703.569339] LustreError: 413623:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11703.594925] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11708.899410] LustreError: 413622:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11708.925940] LustreError: 413622:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11709.832296] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [11714.024952] LustreError: 413617:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11714.061131] LustreError: 413617:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11715.053268] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11717.045994] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11720.170404] LustreError: 413617:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11720.194357] LustreError: 413617:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11723.918496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11724.376978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11728.937774] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [11739.622460] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11739.625073] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11739.634396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11739.649313] Lustre: Skipped 1 previous similar message [11744.738725] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11744.749306] Lustre: Skipped 2 previous similar messages [11745.036245] Lustre: server umount lustre-OST0000 complete [11749.862983] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11749.866790] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11749.879986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11749.890236] Lustre: Skipped 1 previous similar message [11749.908734] Lustre: Skipped 1 previous similar message [11755.014386] LustreError: 413622:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11755.252748] Lustre: server umount lustre-MDT0000 complete [11759.847927] LustreError: 413602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781284477 with bad export cookie 8106815078363182895 [11759.863126] LustreError: 413602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11759.866058] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11760.222516] Lustre: server umount lustre-MDT0001 complete [11771.671588] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [11774.386224] Key type lgssc unregistered [11774.826306] LNet: 416769:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11774.852940] LNetError: 416769:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11774.882253] LNet: Removed LNI 192.168.204.159@tcp [11776.150474] Key type .llcrypt unregistered [11776.156460] Key type ._llcrypt unregistered [11803.848799] Key type ._llcrypt registered [11803.852441] Key type .llcrypt registered [11804.019170] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [11805.330197] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11805.452403] alg: No test for adler32 (adler32-zlib) [11806.559575] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [11806.793386] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [11808.448300] Key type lgssc registered [11810.312289] Lustre: Echo OBD driver; http://www.lustre.org/ [11818.652911] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11818.691554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11834.210091] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11839.329262] LustreError: 418168:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.159@tcp: failed processing log, type 4: rc = -110 [11866.016212] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11871.766782] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [11879.005104] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11885.653910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11886.227116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11886.315595] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11890.560696] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [11891.688649] LustreError: 418193:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11891.705732] LustreError: 418193:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11896.808508] LustreError: 418194:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11898.524089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11898.909023] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11903.650609] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [11911.082451] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11916.734850] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11924.452104] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11924.460392] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11924.478516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11926.518883] Lustre: server umount lustre-OST0000 complete [11932.141869] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [11932.150994] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11932.166978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11937.764247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11937.771282] Lustre: Skipped 2 previous similar messages [11939.908964] Lustre: server umount lustre-MDT0000 complete [11942.900995] LustreError: 419205:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11942.913328] LustreError: 419205:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11943.316261] LustreError: 418174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781284661 with bad export cookie 6595705106039829873 [11943.331877] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11943.338544] LustreError: 418174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11943.702654] Lustre: server umount lustre-MDT0001 complete [11955.428992] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [11958.491606] Key type lgssc unregistered [11958.902614] LNet: 421736:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11958.916282] LNetError: 421736:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11958.945276] LNet: Removed LNI 192.168.204.159@tcp [11959.960136] Key type .llcrypt unregistered [11959.964209] Key type ._llcrypt unregistered [11983.834588] Key type ._llcrypt registered [11983.837143] Key type .llcrypt registered [11983.943638] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [11985.161768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11985.295853] alg: No test for adler32 (adler32-zlib) [11986.359013] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [11986.646919] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [11988.432772] Key type lgssc registered [11989.772871] Lustre: Echo OBD driver; http://www.lustre.org/ [11997.916991] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11997.940599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11999.669367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12004.596972] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [12009.958665] LustreError: 423154:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12009.973651] LustreError: 423154:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12012.315049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12012.693332] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12016.611798] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [12022.634920] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12028.164472] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12035.687806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12036.024845] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12042.360527] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [12044.221360] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12049.399180] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12053.476101] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12053.486811] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12053.496755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12054.499919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12054.501092] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12056.224959] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.59@tcp (stopping) [12056.228317] Lustre: Skipped 2 previous similar messages [12058.841377] Lustre: server umount lustre-MDT0000 complete [12059.619759] LustreError: 423154:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12059.640629] LustreError: 423154:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12061.072189] LustreError: 423134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.59@tcp arrived at 1781284779 with bad export cookie 9827525110874808290 [12061.079814] LustreError: 423134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12062.894429] LustreError: 423136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781284780 with bad export cookie 9827525110874807394 [12062.909886] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12062.915171] LustreError: 423136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12064.737840] LustreError: 423149:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12064.738821] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12064.765877] LustreError: 423149:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12064.783873] Lustre: Skipped 1 previous similar message [12064.788687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12067.747499] LustreError: 423154:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12067.757220] LustreError: 423154:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12069.087104] Lustre: server umount lustre-OST0000 complete [12072.874055] LustreError: 423894:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12072.902981] LustreError: 423894:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12076.236537] Lustre: server umount lustre-MDT0001 complete [12088.042489] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [12091.342446] Key type lgssc unregistered [12091.724846] LNet: 426838:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12091.739920] LNetError: 426838:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12091.755452] LNet: Removed LNI 192.168.204.159@tcp [12092.665995] Key type .llcrypt unregistered [12092.670798] Key type ._llcrypt unregistered [12118.734883] Key type ._llcrypt registered [12118.737750] Key type .llcrypt registered [12118.903395] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [12120.510136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12120.566221] alg: No test for adler32 (adler32-zlib) [12121.675221] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [12121.927447] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [12123.640180] Key type lgssc registered [12125.008030] Lustre: Echo OBD driver; http://www.lustre.org/ [12133.239062] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12133.265472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12134.692285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12139.140921] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [12144.102309] LustreError: 428257:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12144.127135] LustreError: 428257:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12147.342294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12147.908209] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12152.440188] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [12160.069851] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12165.203923] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12171.812125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12172.312253] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12178.731573] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [12179.571399] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12186.386241] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12193.965485] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.59@tcp (stopping) [12194.273411] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12194.286410] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12195.300566] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12195.305421] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12195.314845] Lustre: Skipped 1 previous similar message [12197.600735] Lustre: server umount lustre-MDT0001 complete [12199.062663] LustreError: 428253:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.59@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12200.428554] LustreError: 428258:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12200.444613] LustreError: 428258:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12205.545060] LustreError: 428257:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12205.553779] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12205.589597] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12205.625967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12205.632144] Lustre: Skipped 1 previous similar message [12210.324922] Lustre: server umount lustre-OST0000 complete [12210.658450] LustreError: 428258:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12214.173905] Lustre: server umount lustre-MDT0000 complete [12228.451864] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [12231.126100] Key type lgssc unregistered [12231.494245] LNet: 431941:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12231.499477] LNetError: 431941:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12231.515931] LNet: Removed LNI 192.168.204.159@tcp [12232.380898] Key type .llcrypt unregistered [12232.384187] Key type ._llcrypt unregistered [12255.370677] Key type ._llcrypt registered [12255.374401] Key type .llcrypt registered [12255.469562] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [12269.954762] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [12271.080688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12271.218319] alg: No test for adler32 (adler32-zlib) [12272.257254] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [12272.433381] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [12274.081195] Key type lgssc registered [12275.168870] Lustre: Echo OBD driver; http://www.lustre.org/ [12318.024530] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [12334.055750] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12334.101295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12335.461858] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12335.528774] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12335.643969] Lustre: lustre-MDT0000: new disk, initializing [12335.765483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12335.803627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12340.283711] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [12355.752767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12356.076041] Lustre: lustre-OST0000: new disk, initializing [12356.090537] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12356.103914] Lustre: Skipped 1 previous similar message [12356.117233] Lustre: 436976:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12356.246416] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12362.347664] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [12366.460344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12366.482567] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12366.525463] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12371.463878] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12385.468612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12385.662491] Lustre: 435958:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12385.758940] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12385.832588] Lustre: lustre-MDT0001: new disk, initializing [12385.968724] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12386.006136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12386.016602] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12390.330409] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [12399.902156] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12400.110792] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12405.209751] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12405.459884] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12412.390435] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12412.397506] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12412.425564] Lustre: Skipped 1 previous similar message [12412.447920] Lustre: Skipped 1 previous similar message [12413.117398] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.59@tcp (stopping) [12416.371833] Lustre: server umount lustre-MDT0001 complete [12422.631447] LustreError: 435971:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12422.649146] LustreError: 435971:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [12427.746181] LustreError: 435970:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12427.770732] LustreError: 435970:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12432.873604] LustreError: 435965:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12432.907444] LustreError: 435965:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12437.988088] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12437.995353] LustreError: 435970:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12437.998287] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12438.021468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12443.107384] LustreError: 435971:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12443.109959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12448.231361] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12449.248170] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12449.417126] Lustre: server umount lustre-OST0000 complete [12459.522779] Lustre: server umount lustre-MDT0000 complete [12474.593817] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [12477.312353] Key type lgssc unregistered [12477.595274] LNet: 440727:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12477.606278] LNetError: 440727:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12477.632454] LNet: Removed LNI 192.168.204.159@tcp [12478.658457] Key type .llcrypt unregistered [12478.661300] Key type ._llcrypt unregistered [12517.905695] Key type ._llcrypt registered [12517.908670] Key type .llcrypt registered [12518.029318] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [12518.318368] Key type .llcrypt unregistered [12518.322991] Key type ._llcrypt unregistered [12534.212411] Key type ._llcrypt registered [12534.214263] Key type .llcrypt registered [12534.325534] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [12535.489787] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12535.533361] alg: No test for adler32 (adler32-zlib) [12536.534717] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [12536.800070] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [12538.496188] Key type lgssc registered [12539.703732] Lustre: Echo OBD driver; http://www.lustre.org/ [12565.375495] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12565.430103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12566.629971] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12566.658989] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12566.733534] Lustre: lustre-MDT0000: new disk, initializing [12566.853131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12566.883232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12571.223995] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [12581.721295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12581.867347] Lustre: 444122:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12581.928894] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12581.932849] Lustre: Skipped 1 previous similar message [12582.032709] Lustre: lustre-MDT0001: new disk, initializing [12582.136727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12582.176968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12582.190851] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12587.317343] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [12597.188669] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12602.076957] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12607.519659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12607.724390] Lustre: lustre-OST0000: new disk, initializing [12607.726994] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12607.732407] Lustre: 446090:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12607.789070] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12613.057733] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [12614.208429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12614.223830] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12614.295870] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12620.765246] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12628.449346] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12628.466156] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12628.481259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12629.985790] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12630.002282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12630.012824] Lustre: Skipped 2 previous similar messages [12633.946737] Lustre: server umount lustre-MDT0000 complete [12637.301775] LustreError: 444112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781285355 with bad export cookie 2160819601918121123 [12637.305654] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12637.308925] LustreError: 444112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12637.569849] Lustre: server umount lustre-MDT0001 complete [12650.245950] Lustre: server umount lustre-OST0000 complete [12659.218132] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 13:29:35 (1781285375) [12665.348472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12665.683494] LustreError: 448118:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12665.700690] LustreError: 448118:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [12665.747185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12669.173193] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [12670.950854] LustreError: 448119:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12675.046460] LustreError: 448118:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12677.287253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12677.528487] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12681.984935] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [12689.165652] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12695.031555] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12714.604684] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12714.872238] Lustre: Found index 0 for lustre-OST0000, updating log [12715.151110] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12721.905127] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [12729.823712] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12736.698195] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12744.164707] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12744.173840] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12744.182902] Lustre: Skipped 1 previous similar message [12744.186593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12750.408427] Lustre: server umount lustre-OST0000 complete [12766.690525] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12766.695218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12766.704259] Lustre: Skipped 2 previous similar messages [12766.713440] Lustre: Skipped 3 previous similar messages [12771.811029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12771.817437] Lustre: Skipped 1 previous similar message [12772.570794] Lustre: server umount lustre-MDT0000 complete [12772.836880] LustreError: 448114:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12772.856986] LustreError: 448114:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12774.370950] LNet: 1 local NIs in recovery (showing 1): 192.168.204.159@tcp [12775.841292] LustreError: 448851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781285493 with bad export cookie 2160819601918122516 [12775.849225] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12775.854534] LustreError: 448851:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12776.167982] Lustre: server umount lustre-MDT0001 complete [12784.046796] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 13:31:40 (1781285500) [13399.929487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13400.357323] LustreError: 454549:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13400.373075] LustreError: 454549:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13400.425553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13404.755622] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [13405.675146] LustreError: 454550:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13410.784765] LustreError: 454549:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13412.838535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13413.157883] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13417.352929] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [13424.597699] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13430.216648] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13449.969295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13450.182520] Lustre: Found index 0 for lustre-OST0000, updating log [13450.446168] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13453.280818] LNet: 1 local NIs in recovery (showing 1): 192.168.204.159@tcp [13456.904747] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [13464.897217] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13467.304511] Lustre: server umount lustre-OST0000 complete [13471.786914] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13471.796966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13472.871259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13472.875986] Lustre: Skipped 2 previous similar messages [13474.915943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13474.922062] Lustre: Skipped 3 previous similar messages [13477.224575] Lustre: server umount lustre-MDT0000 complete [13477.990776] LustreError: 454546:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13478.011471] LustreError: 454546:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13480.811303] LustreError: 455283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781286198 with bad export cookie 2160819601918123601 [13480.812284] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13480.821625] LustreError: 455283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13481.068086] Lustre: server umount lustre-MDT0001 complete [13513.761854] LNet: 1 local NIs in recovery (showing 1): 192.168.204.159@tcp [13514.549975] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [13522.973155] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [13557.256498] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 13:44:33 (1781286273) [13577.740035] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [13580.985373] Key type lgssc unregistered [13581.233111] LNet: 461173:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13581.246647] LNetError: 461173:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13711.336117] LNet: Removed LNI 192.168.204.159@tcp [13712.343738] Key type .llcrypt unregistered [13712.347425] Key type ._llcrypt unregistered [13729.352840] Key type ._llcrypt registered [13729.357379] Key type .llcrypt registered [13729.464167] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [13730.395368] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13730.426801] alg: No test for adler32 (adler32-zlib) [13731.551058] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [13731.757081] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [13733.530314] Key type lgssc registered [13734.843570] Lustre: Echo OBD driver; http://www.lustre.org/ [13739.712286] LNet: Added LNI 192.168.204.159@tcp100 [8/256/0/180] [13740.844583] LNet: Added LNI 192.168.204.159@tcp101 [8/256/0/180] [13741.928809] LNet: Added LNI 192.168.204.159@tcp102 [8/256/0/180] [13742.906832] LNet: Added LNI 192.168.204.159@tcp103 [8/256/0/180] [13743.880354] LNet: Added LNI 192.168.204.159@tcp104 [8/256/0/180] [13744.802138] LNet: Added LNI 192.168.204.159@tcp105 [8/256/0/180] [13745.648886] LNet: Added LNI 192.168.204.159@tcp106 [8/256/0/180] [13746.554906] LNet: Added LNI 192.168.204.159@tcp107 [8/256/0/180] [13747.521436] LNet: Added LNI 192.168.204.159@tcp108 [8/256/0/180] [13748.511813] LNet: Added LNI 192.168.204.159@tcp109 [8/256/0/180] [13749.503964] LNet: Added LNI 192.168.204.159@tcp110 [8/256/0/180] [13750.342901] LNet: Added LNI 192.168.204.159@tcp111 [8/256/0/180] [13751.283792] LNet: Added LNI 192.168.204.159@tcp112 [8/256/0/180] [13752.291634] LNet: Added LNI 192.168.204.159@tcp113 [8/256/0/180] [13753.097841] LNet: Added LNI 192.168.204.159@tcp114 [8/256/0/180] [13754.038538] LNet: Added LNI 192.168.204.159@tcp115 [8/256/0/180] [13754.961708] LNet: Added LNI 192.168.204.159@tcp116 [8/256/0/180] [13755.806821] LNet: Added LNI 192.168.204.159@tcp117 [8/256/0/180] [13756.816626] LNet: Added LNI 192.168.204.159@tcp118 [8/256/0/180] [13757.666643] LNet: Added LNI 192.168.204.159@tcp119 [8/256/0/180] [13758.578292] LNet: Added LNI 192.168.204.159@tcp120 [8/256/0/180] [13759.354629] LNet: Added LNI 192.168.204.159@tcp121 [8/256/0/180] [13760.401918] LNet: Added LNI 192.168.204.159@tcp122 [8/256/0/180] [13761.259746] LNet: Added LNI 192.168.204.159@tcp123 [8/256/0/180] [13762.148832] LNet: Added LNI 192.168.204.159@tcp124 [8/256/0/180] [13763.039413] LNet: Added LNI 192.168.204.159@tcp125 [8/256/0/180] [13763.884243] LNet: Added LNI 192.168.204.159@tcp126 [8/256/0/180] [13764.759796] LNet: Added LNI 192.168.204.159@tcp127 [8/256/0/180] [13765.584807] LNet: Added LNI 192.168.204.159@tcp128 [8/256/0/180] [13766.409785] LNet: Added LNI 192.168.204.159@tcp129 [8/256/0/180] [13767.226648] LNet: Added LNI 192.168.204.159@tcp130 [8/256/0/180] [13768.039395] LNet: Added LNI 192.168.204.159@tcp131 [8/256/0/180] [13768.856082] LNet: Added LNI 192.168.204.159@tcp132 [8/256/0/180] [13769.739985] LNet: Added LNI 192.168.204.159@tcp133 [8/256/0/180] [13770.727126] LNet: Added LNI 192.168.204.159@tcp134 [8/256/0/180] [13771.826487] LNet: Added LNI 192.168.204.159@tcp135 [8/256/0/180] [13779.223520] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13779.260417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13780.452800] Lustre: 464209:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.159@tcp: can't add NID 192.168.204.159@tcp130: rc = -75 [13780.574162] LNetError: 461668:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.159 [13780.588677] LNetError: Protocol error connecting to 192.168.204.159@tcp120 at host 192.168.204.159:988: is it running a compatible version of Lustre? [13780.613121] LNetError: Skipped 7 previous similar messages [13780.707702] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13780.756608] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13780.893312] Lustre: lustre-MDT0000: new disk, initializing [13780.896646] LNet: 8 local NIs in recovery (showing 8): 192.168.204.159@tcp121, 192.168.204.159@tcp122, 192.168.204.159@tcp123, 192.168.204.159@tcp124, 192.168.204.159@tcp125, 192.168.204.159@tcp128, 192.168.204.159@tcp120, 192.168.204.159@tcp129 [13780.981605] Lustre: 464265:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.159@tcp: can't add NID 192.168.204.159@tcp130: rc = -75 [13780.993972] Lustre: 464265:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [13781.015775] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13781.049102] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13781.666789] LNetError: 464236:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.159 [13781.675880] LNetError: 464236:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 12 previous similar messages [13781.684161] LNetError: Protocol error connecting to 192.168.204.159@tcp112 at host 192.168.204.159:988: is it running a compatible version of Lustre? [13781.694497] LNetError: Skipped 4 previous similar messages [13783.971929] LNetError: 461669:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.159 [13783.990983] LNetError: 461669:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 8 previous similar messages [13784.004295] LNetError: Protocol error connecting to 192.168.204.159@tcp122 at host 192.168.204.159:988: is it running a compatible version of Lustre? [13784.023803] LNetError: Skipped 11 previous similar messages [13785.669920] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [13795.238957] LNetError: 464233:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.159 [13795.260367] LNetError: 464233:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 3 previous similar messages [13795.272680] LNetError: Protocol error connecting to 192.168.204.159@tcp122 at host 192.168.204.159:988: is it running a compatible version of Lustre? [13796.101833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13796.294204] Lustre: 464226:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13796.347383] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13796.351025] Lustre: Skipped 1 previous similar message [13796.440470] Lustre: lustre-MDT0001: new disk, initializing [13796.477905] Lustre: 465026:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.159@tcp: can't add NID 192.168.204.159@tcp130: rc = -75 [13796.502349] Lustre: 465026:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [13796.634896] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13796.678485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13796.704504] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13801.876780] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [13805.158865] Lustre: 465391:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.159@tcp: can't add NID 192.168.204.159@tcp130: rc = -75 [13805.168759] Lustre: 465391:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [13811.639806] LNetError: 464229:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.159 [13811.659735] LNetError: Protocol error connecting to 192.168.204.159@tcp122 at host 192.168.204.159:988: is it running a compatible version of Lustre? [13813.091941] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13818.617341] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13819.809965] LNetError: 464229:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.159 [13819.822328] LNetError: 464229:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 7 previous similar messages [13819.830360] LNetError: Protocol error connecting to 192.168.204.159@tcp128 at host 192.168.204.159:988: is it running a compatible version of Lustre? [13819.839786] LNetError: Skipped 7 previous similar messages [13824.894419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13825.249393] Lustre: lustre-OST0000: new disk, initializing [13825.254315] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13825.266230] Lustre: 466202:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13825.370741] Lustre: 466207:0:(lustre_peer.c:105:class_add_uuid()) 192.168.204.159@tcp: can't add NID 192.168.204.159@tcp130: rc = -75 [13825.381436] Lustre: 466207:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [13825.431560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13831.130406] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [13831.748551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13831.754938] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13831.897642] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13837.217585] LNetError: 461668:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.159 [13837.241617] LNetError: 461668:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 10 previous similar messages [13837.251157] LNetError: Protocol error connecting to 192.168.204.159@tcp119 at host 192.168.204.159:988: is it running a compatible version of Lustre? [13837.266360] LNetError: Skipped 10 previous similar messages [13840.289144] LNet: 1 local NIs in recovery (showing 1): 192.168.204.159@tcp122 [13841.371949] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13852.646059] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13852.649767] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13852.653404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13852.659153] LustreError: Skipped 1 previous similar message [13852.680178] Lustre: Skipped 1 previous similar message [13857.761797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13857.780241] Lustre: Skipped 1 previous similar message [13862.880135] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13862.887484] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13862.902814] Lustre: Skipped 1 previous similar message [13863.080081] Lustre: server umount lustre-OST0000 complete [13868.003339] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13868.021412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13868.024291] Lustre: Skipped 2 previous similar messages [13873.121315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13873.135231] Lustre: Skipped 1 previous similar message [13876.133082] LNetError: 461668:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.159 [13876.154393] LNetError: 461668:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 18 previous similar messages [13876.171123] LNetError: Protocol error connecting to 192.168.204.159@tcp127 at host 192.168.204.159:988: is it running a compatible version of Lustre? [13876.191204] LNetError: Skipped 18 previous similar messages [13881.318064] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13881.562404] Lustre: server umount lustre-MDT0000 complete [13885.660329] LustreError: 464219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781286603 with bad export cookie 5614256343349865201 [13885.672392] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13885.676464] LustreError: 464219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13886.328723] Lustre: server umount lustre-MDT0001 complete [13897.239899] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [13900.120170] Key type lgssc unregistered [13900.530398] LNet: 468116:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13900.544227] LNetError: 468116:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13900.557501] LNet: Removed LNI 192.168.204.159@tcp [13901.600446] LNet: Removed LNI 192.168.204.159@tcp100 [13901.606237] LNetError: 468116:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13901.620784] LNetError: 468116:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13901.630312] LNet: Removed LNI 192.168.204.159@tcp101 [13902.688737] LNet: Removed LNI 192.168.204.159@tcp102 [13902.691596] LNetError: 468116:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13902.716561] LNetError: 468116:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13902.741367] LNet: Removed LNI 192.168.204.159@tcp103 [13903.778673] LNet: Removed LNI 192.168.204.159@tcp104 [13904.802598] LNet: Removed LNI 192.168.204.159@tcp105 [13904.804580] LNetError: 468116:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13904.813603] LNetError: 468116:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [13904.818935] LNet: Removed LNI 192.168.204.159@tcp106 [13905.824588] LNet: Removed LNI 192.168.204.159@tcp107 [13906.848214] LNet: Removed LNI 192.168.204.159@tcp108 [13907.872228] LNet: Removed LNI 192.168.204.159@tcp109 [13908.896210] LNet: Removed LNI 192.168.204.159@tcp110 [13908.903642] LNetError: 468116:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13908.922697] LNetError: 468116:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [13908.932494] LNet: Removed LNI 192.168.204.159@tcp111 [13909.984376] LNet: Removed LNI 192.168.204.159@tcp112 [13911.008285] LNet: Removed LNI 192.168.204.159@tcp113 [13912.033983] LNet: Removed LNI 192.168.204.159@tcp114 [13913.056153] LNet: Removed LNI 192.168.204.159@tcp115 [13914.080527] LNet: Removed LNI 192.168.204.159@tcp116 [13915.104154] LNet: Removed LNI 192.168.204.159@tcp117 [13916.192227] LNet: Removed LNI 192.168.204.159@tcp118 [13917.219253] LNet: Removed LNI 192.168.204.159@tcp119 [13917.222489] LNetError: 468116:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13917.231614] LNetError: 468116:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [13917.249732] LNet: Removed LNI 192.168.204.159@tcp120 [13918.304411] LNet: Removed LNI 192.168.204.159@tcp121 [13918.308690] LNet: Removed LNI 192.168.204.159@tcp122 [13919.328150] LNet: Removed LNI 192.168.204.159@tcp123 [13920.352368] LNet: Removed LNI 192.168.204.159@tcp124 [13921.376143] LNet: Removed LNI 192.168.204.159@tcp125 [13922.400399] LNet: Removed LNI 192.168.204.159@tcp126 [13923.424167] LNet: Removed LNI 192.168.204.159@tcp127 [13924.448168] LNet: Removed LNI 192.168.204.159@tcp128 [13925.472164] LNet: Removed LNI 192.168.204.159@tcp129 [13925.477171] LNet: Removed LNI 192.168.204.159@tcp130 [13925.489486] LNet: Removed LNI 192.168.204.159@tcp131 [13925.496162] LNet: Removed LNI 192.168.204.159@tcp132 [13925.506455] LNet: Removed LNI 192.168.204.159@tcp133 [13925.512166] LNet: Removed LNI 192.168.204.159@tcp134 [13925.526483] LNet: Removed LNI 192.168.204.159@tcp135 [13926.422282] Key type .llcrypt unregistered [13926.425129] Key type ._llcrypt unregistered [13950.205656] Key type ._llcrypt registered [13950.207893] Key type .llcrypt registered [13950.299823] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [13964.612559] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [13965.612412] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13965.724158] alg: No test for adler32 (adler32-zlib) [13966.838154] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [13967.131636] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [13968.816210] Key type lgssc registered [13969.919623] Lustre: Echo OBD driver; http://www.lustre.org/ [14000.755888] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 13:51:56 (1781286716) [14016.887232] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [14019.257573] Key type lgssc unregistered [14019.536307] LNet: 472204:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14019.541089] LNetError: 472204:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14019.556456] LNet: Removed LNI 192.168.204.159@tcp [14020.226391] Key type .llcrypt unregistered [14020.229482] Key type ._llcrypt unregistered [14032.758629] Key type ._llcrypt registered [14032.760822] Key type .llcrypt registered [14032.864594] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [14034.174942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14034.229772] alg: No test for adler32 (adler32-zlib) [14035.342637] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [14035.609940] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [14037.289113] Key type lgssc registered [14038.664560] Lustre: Echo OBD driver; http://www.lustre.org/ [14047.733685] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14047.786684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14049.061202] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14049.079893] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14049.129479] Lustre: lustre-MDT0000: new disk, initializing [14049.220573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14049.250065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14054.165706] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [14064.829028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14064.930225] Lustre: 473398:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14065.014144] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14065.025459] Lustre: Skipped 1 previous similar message [14065.098253] Lustre: lustre-MDT0001: new disk, initializing [14065.196620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14065.221093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14065.229727] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14069.778360] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [14078.324681] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14082.845417] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14088.530321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14088.756454] Lustre: lustre-OST0000: new disk, initializing [14088.759183] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14088.766518] Lustre: 475365:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14088.845854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14093.467596] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [14096.445563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14096.455885] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14096.546165] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14100.204576] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14103.933685] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing lnet_if_list [14108.839550] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14108.925633] LNet: Added LNI 192.168.204.159@tcp10 [8/256/0/180] [14115.170403] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14115.220374] LNet: Added LNI 192.168.204.159@tcp11 [8/256/0/180] [14120.914092] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14120.938516] LNet: Added LNI 192.168.204.159@tcp12 [8/256/0/180] [14125.347351] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14125.380274] LNet: Added LNI 192.168.204.159@tcp13 [8/256/0/180] [14129.984874] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14130.030048] LNet: Added LNI 192.168.204.159@tcp14 [8/256/0/180] [14135.578436] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14135.614421] LNet: Added LNI 192.168.204.159@tcp15 [8/256/0/180] [14140.888590] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14140.926750] LNet: Added LNI 192.168.204.159@tcp16 [8/256/0/180] [14145.441555] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14145.485171] LNet: Added LNI 192.168.204.159@tcp17 [8/256/0/180] [14150.131761] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14150.161072] LNet: Added LNI 192.168.204.159@tcp18 [8/256/0/180] [14154.320761] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14154.365813] LNet: Added LNI 192.168.204.159@tcp19 [8/256/0/180] [14159.855686] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14159.906513] LNet: Added LNI 192.168.204.159@tcp20 [8/256/0/180] [14164.977226] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14165.024836] LNet: Added LNI 192.168.204.159@tcp21 [8/256/0/180] [14169.797705] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14169.842534] LNet: Added LNI 192.168.204.159@tcp22 [8/256/0/180] [14174.466661] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14174.485730] LNet: Added LNI 192.168.204.159@tcp23 [8/256/0/180] [14179.403928] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14179.439495] LNet: Added LNI 192.168.204.159@tcp24 [8/256/0/180] [14184.401371] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14184.443844] LNet: Added LNI 192.168.204.159@tcp25 [8/256/0/180] [14189.621912] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14189.644800] LNet: Added LNI 192.168.204.159@tcp26 [8/256/0/180] [14195.012599] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14195.035293] LNet: Added LNI 192.168.204.159@tcp27 [8/256/0/180] [14200.564273] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14200.598189] LNet: Added LNI 192.168.204.159@tcp28 [8/256/0/180] [14206.912824] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14206.953372] LNet: Added LNI 192.168.204.159@tcp29 [8/256/0/180] [14212.575646] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14212.603539] LNet: Added LNI 192.168.204.159@tcp30 [8/256/0/180] [14217.559731] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14217.581619] LNet: Added LNI 192.168.204.159@tcp31 [8/256/0/180] [14222.935549] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14222.957394] LNet: Added LNI 192.168.204.159@tcp32 [8/256/0/180] [14227.925183] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14227.954666] LNet: Added LNI 192.168.204.159@tcp33 [8/256/0/180] [14233.922780] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14233.953761] LNet: Added LNI 192.168.204.159@tcp34 [8/256/0/180] [14239.853636] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14239.912280] LNet: Added LNI 192.168.204.159@tcp35 [8/256/0/180] [14246.631652] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14246.662634] LNet: Added LNI 192.168.204.159@tcp36 [8/256/0/180] [14252.889641] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14252.929403] LNet: Added LNI 192.168.204.159@tcp37 [8/256/0/180] [14259.728549] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14259.768823] LNet: Added LNI 192.168.204.159@tcp38 [8/256/0/180] [14266.157627] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14266.192355] LNet: Added LNI 192.168.204.159@tcp39 [8/256/0/180] [14273.169475] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14273.198206] LNet: Added LNI 192.168.204.159@tcp40 [8/256/0/180] [14280.449131] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14280.499512] LNet: Added LNI 192.168.204.159@tcp41 [8/256/0/180] [14288.434887] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14288.462048] LNet: Added LNI 192.168.204.159@tcp42 [8/256/0/180] [14295.584932] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14295.612176] LNet: Added LNI 192.168.204.159@tcp43 [8/256/0/180] [14304.552904] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14304.588609] LNet: Added LNI 192.168.204.159@tcp44 [8/256/0/180] [14311.529361] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14311.581338] LNet: Added LNI 192.168.204.159@tcp45 [8/256/0/180] [14318.536535] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14318.561217] LNet: Added LNI 192.168.204.159@tcp46 [8/256/0/180] [14327.300728] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14327.328924] LNet: Added LNI 192.168.204.159@tcp47 [8/256/0/180] [14334.800742] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14334.827523] LNet: Added LNI 192.168.204.159@tcp48 [8/256/0/180] [14343.385944] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14343.410313] LNet: Added LNI 192.168.204.159@tcp49 [8/256/0/180] [14351.380930] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14351.406705] LNet: Added LNI 192.168.204.159@tcp50 [8/256/0/180] [14377.444507] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14377.452360] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14377.483692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14378.472965] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14378.478106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14378.491138] Lustre: Skipped 1 previous similar message [14378.498044] Lustre: Skipped 1 previous similar message [14380.704730] Lustre: server umount lustre-MDT0000 complete [14384.690124] LustreError: 473389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781287102 with bad export cookie 5724620690965059690 [14384.695498] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14384.706296] LustreError: 473389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14385.053815] Lustre: server umount lustre-MDT0001 complete [14404.066122] Lustre: 472669:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781287106/real 1781287106] req@00000000fc640c3d x1867814538475136/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781287122 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14404.107050] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14405.166663] Lustre: 472669:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781287107/real 1781287107] req@000000008e1366a1 x1867814538475264/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781287123 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14408.119971] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [14409.184100] Lustre: 472671:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781287111/real 1781287111] req@00000000163c54e8 x1867814538475392/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781287127 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14415.328740] Lustre: 472668:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781287117/real 1781287117] req@0000000075aada52 x1867814538475776/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781287133 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14415.361810] Lustre: 472668:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [14419.944833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14429.730083] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x4f71f03026d2e250 [14429.762174] Lustre: MGC192.168.204.159@tcp: Connection restored to 0@lo (at 0@lo) [14430.218688] LustreError: 475358:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14430.238033] LustreError: 475358:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [14430.322836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14433.386433] LustreError: 475359:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14433.392114] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14434.401680] LustreError: 475358:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14434.817318] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [14439.911458] LustreError: 475359:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14442.460506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14442.831278] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14446.791949] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [14449.158929] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14454.544207] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14459.841259] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14478.824862] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14478.841777] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14478.855740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14478.860669] Lustre: Skipped 1 previous similar message [14482.756610] Lustre: server umount lustre-OST0000 complete [14489.057703] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14489.066900] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14489.082554] Lustre: Skipped 1 previous similar message [14489.089574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14489.097723] Lustre: Skipped 2 previous similar messages [14494.348335] Lustre: server umount lustre-MDT0000 complete [14495.206908] LustreError: 499630:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14495.228239] LustreError: 499630:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [14497.527297] LustreError: 473390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781287215 with bad export cookie 5724620690965062224 [14497.528557] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14497.534393] LustreError: 473390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14497.795664] Lustre: server umount lustre-MDT0001 complete [14507.878852] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [14510.258269] Key type lgssc unregistered [14510.596441] LNet: 502405:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14510.612129] LNetError: 502405:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14510.634240] LNet: Removed LNI 192.168.204.159@tcp [14510.639160] LNet: Removed LNI 192.168.204.159@tcp10 [14510.644097] LNet: Removed LNI 192.168.204.159@tcp11 [14510.646324] LNet: Removed LNI 192.168.204.159@tcp12 [14510.648716] LNet: Removed LNI 192.168.204.159@tcp13 [14510.651386] LNet: Removed LNI 192.168.204.159@tcp14 [14510.653866] LNet: Removed LNI 192.168.204.159@tcp15 [14510.656436] LNet: Removed LNI 192.168.204.159@tcp16 [14510.659280] LNet: Removed LNI 192.168.204.159@tcp17 [14510.662291] LNet: Removed LNI 192.168.204.159@tcp18 [14510.667946] LNet: Removed LNI 192.168.204.159@tcp19 [14510.673228] LNet: Removed LNI 192.168.204.159@tcp20 [14510.677359] LNet: Removed LNI 192.168.204.159@tcp21 [14510.681262] LNet: Removed LNI 192.168.204.159@tcp22 [14510.685561] LNet: Removed LNI 192.168.204.159@tcp23 [14510.689320] LNet: Removed LNI 192.168.204.159@tcp24 [14510.692224] LNet: Removed LNI 192.168.204.159@tcp25 [14510.695407] LNet: Removed LNI 192.168.204.159@tcp26 [14510.698265] LNet: Removed LNI 192.168.204.159@tcp27 [14510.700670] LNet: Removed LNI 192.168.204.159@tcp28 [14510.704543] LNet: Removed LNI 192.168.204.159@tcp29 [14510.707329] LNet: Removed LNI 192.168.204.159@tcp30 [14510.709583] LNet: Removed LNI 192.168.204.159@tcp31 [14510.711769] LNet: Removed LNI 192.168.204.159@tcp32 [14510.714916] LNet: Removed LNI 192.168.204.159@tcp33 [14510.717953] LNet: Removed LNI 192.168.204.159@tcp34 [14510.720670] LNet: Removed LNI 192.168.204.159@tcp35 [14510.724959] LNet: Removed LNI 192.168.204.159@tcp36 [14510.728578] LNet: Removed LNI 192.168.204.159@tcp37 [14510.733458] LNet: Removed LNI 192.168.204.159@tcp38 [14510.736951] LNet: Removed LNI 192.168.204.159@tcp39 [14510.739616] LNet: Removed LNI 192.168.204.159@tcp40 [14510.745172] LNet: Removed LNI 192.168.204.159@tcp41 [14510.749536] LNet: Removed LNI 192.168.204.159@tcp42 [14510.753729] LNet: Removed LNI 192.168.204.159@tcp43 [14510.756629] LNet: Removed LNI 192.168.204.159@tcp44 [14510.759239] LNet: Removed LNI 192.168.204.159@tcp45 [14510.761661] LNet: Removed LNI 192.168.204.159@tcp46 [14510.764629] LNet: Removed LNI 192.168.204.159@tcp47 [14510.767123] LNet: Removed LNI 192.168.204.159@tcp48 [14510.768974] LNet: Removed LNI 192.168.204.159@tcp49 [14510.778060] LNet: Removed LNI 192.168.204.159@tcp50 [14511.402174] Key type .llcrypt unregistered [14511.404347] Key type ._llcrypt unregistered [14530.295463] Key type ._llcrypt registered [14530.298120] Key type .llcrypt registered [14530.415211] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [14541.557952] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [14542.483902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14542.503711] alg: No test for adler32 (adler32-zlib) [14543.697985] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [14543.945217] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [14545.632145] Key type lgssc registered [14546.570665] Lustre: Echo OBD driver; http://www.lustre.org/ [14574.401649] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14574.429517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14575.694660] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14575.736875] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14575.814407] Lustre: lustre-MDT0000: new disk, initializing [14575.938522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14575.954558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14580.100559] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [14589.529987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14589.657956] Lustre: 505589:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14589.731843] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14589.742207] Lustre: Skipped 1 previous similar message [14589.832777] Lustre: lustre-MDT0001: new disk, initializing [14589.973180] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14590.031624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14590.052633] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14594.467088] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [14605.787969] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14611.068139] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14618.122732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14618.410720] Lustre: *** cfs_fail_loc=90f, val=0*** [14618.526959] Lustre: lustre-OST0000: new disk, initializing [14618.531529] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14618.536813] Lustre: 507606:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14618.609127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14623.764439] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [14626.383666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14626.390699] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14626.479190] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14632.495975] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14642.149779] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14642.158365] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14642.165605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14642.180677] Lustre: Skipped 1 previous similar message [14642.187416] Lustre: Skipped 1 previous similar message [14647.271297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14647.281734] Lustre: Skipped 1 previous similar message [14652.397454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14652.403969] Lustre: Skipped 1 previous similar message [14653.408161] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14653.539190] Lustre: server umount lustre-OST0000 complete [14657.506182] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14657.528989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14657.538540] Lustre: Skipped 1 previous similar message [14662.627972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14662.638297] Lustre: Skipped 1 previous similar message [14663.728516] Lustre: server umount lustre-MDT0000 complete [14667.671748] LustreError: 505582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781287385 with bad export cookie 311855530483015339 [14667.672736] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14667.678109] LustreError: 505582:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14668.055578] Lustre: server umount lustre-MDT0001 complete [14678.908709] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [14681.405707] Key type lgssc unregistered [14681.736633] LNet: 509511:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14681.744891] LNetError: 509511:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14681.771880] LNet: Removed LNI 192.168.204.159@tcp [14682.527484] Key type .llcrypt unregistered [14682.529628] Key type ._llcrypt unregistered [14702.968133] Key type ._llcrypt registered [14702.971474] Key type .llcrypt registered [14703.074348] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [14704.613488] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14704.702943] alg: No test for adler32 (adler32-zlib) [14706.096610] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [14706.506490] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [14708.313227] Key type lgssc registered [14710.014214] Lustre: Echo OBD driver; http://www.lustre.org/ [14718.407209] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14718.444379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14719.944136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14725.045140] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [14729.191425] LustreError: 510788:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14729.225834] LustreError: 510788:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [14733.357893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14733.650590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14738.057611] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [14745.476989] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14750.497049] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14757.483628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14757.881343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14764.381698] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [14772.407557] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14784.345255] loop0: detected capacity change from 0 to 204800000 [14784.611466] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14784.641837] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14784.660973] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14792.019906] loop0: detected capacity change from 0 to 204800000 [14792.305854] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14792.325914] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14792.340349] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14798.898630] loop0: detected capacity change from 0 to 204800000 [14804.800639] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14805.000663] Lustre: fs15246-OST0000: new disk, initializing [14805.005754] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [14805.014828] Lustre: 514379:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14805.070783] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14811.222507] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [14820.663622] loop1: detected capacity change from 0 to 204800000 [14826.629675] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14826.737228] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [14826.776547] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [14826.843934] Lustre: fs15246-MDT0000: new disk, initializing [14827.066725] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14827.086371] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14827.198428] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [14828.257184] Lustre: 515501:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@0000000035401953 x1867815241956864/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1781287562 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [14831.796894] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [14837.385112] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14837.405659] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14837.440375] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14855.666381] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14855.697211] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14860.788460] LustreError: 510782:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14860.900175] Lustre: server umount fs15246-MDT0000 complete [14865.896238] LustreError: 510784:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14870.460654] Lustre: server umount fs15246-OST0000 complete [14886.371380] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14886.377481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14886.392169] Lustre: Skipped 1 previous similar message [14886.402350] Lustre: Skipped 1 previous similar message [14891.491185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14896.612949] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14896.627028] Lustre: Skipped 2 previous similar messages [14897.122543] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14897.316908] Lustre: server umount lustre-OST0000 complete [14901.737798] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14901.741239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14901.751239] Lustre: Skipped 1 previous similar message [14901.765055] Lustre: Skipped 1 previous similar message [14907.164388] Lustre: server umount lustre-MDT0000 complete [14910.664359] LustreError: 510767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781287628 with bad export cookie 9256161982804756306 [14910.674556] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14910.999523] Lustre: server umount lustre-MDT0001 complete [14921.188501] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [14923.970237] Key type lgssc unregistered [14924.272116] LNet: 518601:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14924.276357] LNetError: 518601:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14924.296862] LNet: Removed LNI 192.168.204.159@tcp [14925.021152] Key type .llcrypt unregistered [14925.024368] Key type ._llcrypt unregistered [14941.352528] Key type ._llcrypt registered [14941.356528] Key type .llcrypt registered [14941.466673] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [14942.273788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14942.392561] alg: No test for adler32 (adler32-zlib) [14943.454990] Lustre: Lustre: Build Version: 2.17.54_4_gd2567f2 [14943.644584] LNet: Added LNI 192.168.204.159@tcp [8/256/0/180] [14945.312626] Key type lgssc registered [14946.392327] Lustre: Echo OBD driver; http://www.lustre.org/ [14983.622254] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [14994.185489] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [15028.249658] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15028.286247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15029.546490] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15029.571670] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15029.628247] Lustre: lustre-MDT0000: new disk, initializing [15029.718119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15029.741558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15033.746835] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [15043.601392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15043.704910] Lustre: 523061:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15043.743807] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15043.748209] Lustre: Skipped 1 previous similar message [15043.791380] Lustre: lustre-MDT0001: new disk, initializing [15043.874596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15043.912117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15043.921497] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15048.222311] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [15056.785428] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [15062.214579] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [15068.075277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15068.313469] Lustre: lustre-OST0000: new disk, initializing [15068.320291] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15068.328641] Lustre: 525031:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15068.393336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15074.082209] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_default_debug -1 all [15076.930238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15076.936802] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15077.059526] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15081.732885] Lustre: DEBUG MARKER: oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [15085.459821] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15085.619961] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15089.375742] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15089.571418] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15092.708515] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15092.713554] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15092.727851] LustreError: Skipped 1 previous similar message [15092.733819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15092.748707] Lustre: Skipped 1 previous similar message [15097.051506] Lustre: server umount lustre-OST0000 complete [15100.387090] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15100.389955] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15100.400780] Lustre: Skipped 1 previous similar message [15100.406261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15102.950447] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15102.951664] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15102.974452] Lustre: Skipped 1 previous similar message [15106.651758] Lustre: server umount lustre-MDT0000 complete [15109.935903] LustreError: 523052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781287827 with bad export cookie 12376608880666449822 [15109.946989] LustreError: MGC192.168.204.159@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15109.957504] LustreError: 523052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15110.241290] Lustre: server umount lustre-MDT0001 complete [15133.177795] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing set_hostid [15144.046762] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing load_modules_local [15176.606632] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14909 sec ========= 14:11:31 (1781287891) [15178.868375] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 14:11:34 (1781287894) === [15183.238576] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 14:11:38 (1781287898) === [15227.006396] Lustre: DEBUG MARKER: oleg459-server.virtnet: executing unload_modules_local [15229.900689] Key type lgssc unregistered [15230.168342] LNet: 531672:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15230.181344] LNetError: 531672:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [15230.197364] LNet: Removed LNI 192.168.204.159@tcp [15231.016309] Key type .llcrypt unregistered [15231.018331] Key type ._llcrypt unregistered