[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 495218902 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.003271] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005018] kvm-guest: setup PV IPIs [ 0.007894] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009016] pid_max: default: 32768 minimum: 301 [ 0.010145] LSM: Security Framework initializing [ 0.011069] Yama: becoming mindful. [ 0.012037] SELinux: Initializing. [ 0.013088] *** VALIDATE selinux *** [ 0.021751] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026745] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028072] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030150] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032106] *** VALIDATE tmpfs *** [ 0.033468] *** VALIDATE proc *** [ 0.034250] *** VALIDATE cgroup *** [ 0.035014] *** VALIDATE cgroup2 *** [ 0.036304] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038116] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039015] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040037] Spectre V2 : User space: Vulnerable [ 0.041011] Speculative Store Bypass: Vulnerable [ 0.043839] debug: unmapping init [mem 0xffffffff8e659000-0xffffffff8e660fff] [ 0.045175] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046828] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047020] ... version: 2 [ 0.048014] ... bit width: 48 [ 0.049013] ... generic registers: 4 [ 0.050015] ... value mask: 0000ffffffffffff [ 0.051019] ... max period: 00007fffffffffff [ 0.052018] ... fixed-purpose events: 3 [ 0.053012] ... event mask: 000000070000000f [ 0.054375] rcu: Hierarchical SRCU implementation. [ 0.056522] smp: Bringing up secondary CPUs ... [ 0.057624] x86: Booting SMP configuration: [ 0.058033] .... node #0, CPUs: #1 #2 #3 [ 0.062397] smp: Brought up 1 node, 4 CPUs [ 0.064020] smpboot: Max logical packages: 1 [ 0.065020] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.237027] node 0 deferred pages initialised in 169ms [ 0.241015] devtmpfs: initialized [ 0.242136] x86/mm: Memory block size: 128MB [ 0.244461] gcov: version magic: 0x41383552 [ 0.246258] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.247097] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.248279] pinctrl core: initialized pinctrl subsystem [ 0.249212] [ 0.249660] ************************************************************* [ 0.250014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.251012] ** ** [ 0.252011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.253012] ** ** [ 0.254017] ** This means that this kernel is built to expose internal ** [ 0.255017] ** IOMMU data structures, which may compromise security on ** [ 0.256015] ** your system. ** [ 0.257019] ** ** [ 0.258014] ** If you see this message and you are not debugging the ** [ 0.259015] ** kernel, report this immediately to your vendor! ** [ 0.260014] ** ** [ 0.261015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.262017] ************************************************************* [ 0.263630] NET: Registered protocol family 16 [ 0.264536] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.265071] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.266069] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.268059] cpuidle: using governor menu [ 0.269811] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.272481] PCI: Using configuration type 1 for base access [ 0.275139] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.285063] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.286020] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.288089] cryptd: max_cpu_qlen set to 1000 [ 0.290322] ACPI: Added _OSI(Module Device) [ 0.291017] ACPI: Added _OSI(Processor Device) [ 0.292014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.293015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.296040] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.303340] ACPI: Interpreter enabled [ 0.304000] ACPI: PM: (supports S0 S3 S4 S5) [ 0.306029] ACPI: Using IOAPIC for interrupt routing [ 0.308142] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.311495] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.322885] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.323000] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.325033] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.328120] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.333289] acpiphp: Slot [2] registered [ 0.335250] acpiphp: Slot [3] registered [ 0.337229] acpiphp: Slot [4] registered [ 0.338111] acpiphp: Slot [5] registered [ 0.340149] acpiphp: Slot [6] registered [ 0.341000] acpiphp: Slot [7] registered [ 0.341000] acpiphp: Slot [8] registered [ 0.343113] acpiphp: Slot [9] registered [ 0.344128] acpiphp: Slot [10] registered [ 0.346105] acpiphp: Slot [11] registered [ 0.347112] acpiphp: Slot [12] registered [ 0.349108] acpiphp: Slot [13] registered [ 0.350132] acpiphp: Slot [14] registered [ 0.352104] acpiphp: Slot [15] registered [ 0.353138] acpiphp: Slot [16] registered [ 0.355095] acpiphp: Slot [17] registered [ 0.356096] acpiphp: Slot [18] registered [ 0.357107] acpiphp: Slot [19] registered [ 0.359111] acpiphp: Slot [20] registered [ 0.360102] acpiphp: Slot [21] registered [ 0.361105] acpiphp: Slot [22] registered [ 0.363157] acpiphp: Slot [23] registered [ 0.364133] acpiphp: Slot [24] registered [ 0.366107] acpiphp: Slot [25] registered [ 0.367100] acpiphp: Slot [26] registered [ 0.368131] acpiphp: Slot [27] registered [ 0.369101] acpiphp: Slot [28] registered [ 0.371132] acpiphp: Slot [29] registered [ 0.372092] acpiphp: Slot [30] registered [ 0.373086] acpiphp: Slot [31] registered [ 0.375095] PCI host bridge to bus 0000:00 [ 0.376024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.378025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.381023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.383023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.386028] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.388040] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.391230] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.396233] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.401789] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.411030] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.416548] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.419048] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.421030] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.424044] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.428561] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.430957] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.432053] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.434725] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.440020] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.453019] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.459019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.464664] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.470021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.475028] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.493030] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.502876] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.510028] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.517020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.531032] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.541972] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.553024] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.560028] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.579025] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.593880] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.600028] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.607018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.620021] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.628662] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.634019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.639018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.658023] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.668993] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.676034] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.682031] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.699030] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.710530] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.713479] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.716522] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.718437] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.721288] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.727050] iommu: Default domain type: Passthrough [ 0.728637] SCSI subsystem initialized [ 0.729217] ACPI: bus type USB registered [ 0.730171] usbcore: registered new interface driver usbfs [ 0.731170] usbcore: registered new interface driver hub [ 0.733099] usbcore: registered new device driver usb [ 0.735220] pps_core: LinuxPPS API ver. 1 registered [ 0.737021] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.740089] PTP clock support registered [ 0.742234] EDAC MC: Ver: 3.0.0 [ 0.746817] PCI: Using ACPI for IRQ routing [ 0.749013] NetLabel: Initializing [ 0.750009] NetLabel: domain hash size = 128 [ 0.752016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.754137] NetLabel: unlabeled traffic allowed by default [ 0.756192] vgaarb: loaded [ 0.758303] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.760019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.768425] clocksource: Switched to clocksource kvm-clock [ 0.885071] VFS: Disk quotas dquot_6.6.0 [ 0.887153] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.890331] *** VALIDATE ramfs *** [ 0.891910] *** VALIDATE hugetlbfs *** [ 0.893978] pnp: PnP ACPI init [ 0.896652] pnp: PnP ACPI: found 6 devices [ 0.915293] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.919078] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.921644] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.924015] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.926830] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.929358] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.931965] NET: Registered protocol family 2 [ 0.934646] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.940153] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.944205] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.950562] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.955137] TCP: Hash tables configured (established 65536 bind 65536) [ 0.958138] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.962090] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.964955] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.968067] NET: Registered protocol family 1 [ 0.971327] RPC: Registered named UNIX socket transport module. [ 0.973754] RPC: Registered udp transport module. [ 0.975529] RPC: Registered tcp transport module. [ 0.977326] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.979823] NET: Registered protocol family 44 [ 0.981777] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.984253] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.986556] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.989194] PCI: CLS 0 bytes, default 64 [ 0.991604] Unpacking initramfs... [ 2.462366] debug: unmapping init [mem 0xffff9e2abcc54000-0xffff9e2abffbffff] [ 2.468136] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.470096] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.473254] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.961134] Initialise system trusted keyrings [ 2.962776] Key type blacklist registered [ 2.964608] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.975599] zbud: loaded [ 2.979411] *** VALIDATE nfs *** [ 2.980524] *** VALIDATE nfs4 *** [ 2.982177] pstore: using deflate compression [ 2.985945] Platform Keyring initialized [ 3.099619] NET: Registered protocol family 38 [ 3.101102] Key type asymmetric registered [ 3.102369] Asymmetric key parser 'x509' registered [ 3.103889] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.109871] io scheduler mq-deadline registered [ 3.111863] io scheduler kyber registered [ 3.118681] io scheduler bfq registered [ 3.120711] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.124077] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.127059] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.130187] ACPI: Power Button [PWRF] [ 3.229370] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.324808] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.514954] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.619321] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.820703] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.850787] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.880356] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.885849] Non-volatile memory driver v1.3 [ 3.888135] Linux agpgart interface v0.103 [ 3.920425] virtio_blk virtio1: [vda] 133848 512-byte logical blocks (68.5 MB/65.4 MiB) [ 3.925565] vda: detected capacity change from 0 to 68530176 [ 3.942233] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.945487] vdb: detected capacity change from 0 to 1073741824 [ 3.960591] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.963348] vdc: detected capacity change from 0 to 2621440000 [ 3.979525] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.982838] vdd: detected capacity change from 0 to 2621440000 [ 3.998634] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.001793] vde: detected capacity change from 0 to 4294967296 [ 4.017956] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.021155] vdf: detected capacity change from 0 to 4294967296 [ 4.028222] libphy: Fixed MDIO Bus: probed [ 4.037700] usbcore: registered new interface driver usbserial_generic [ 4.040258] usbserial: USB Serial support registered for generic [ 4.042768] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.047380] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.049545] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.052143] mousedev: PS/2 mouse device common for all mice [ 4.055128] rtc_cmos 00:05: RTC can wake from S4 [ 4.058265] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.061700] rtc_cmos 00:05: registered as rtc0 [ 4.064733] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.066416] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.068216] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.074960] intel_pstate: CPU model not supported [ 4.080762] hid: raw HID events driver (C) Jiri Kosina [ 4.083039] usbcore: registered new interface driver usbhid [ 4.085223] usbhid: USB HID core driver [ 4.086751] drop_monitor: Initializing network drop monitor service [ 4.089307] Initializing XFRM netlink socket [ 4.091403] NET: Registered protocol family 10 [ 4.096172] Segment Routing with IPv6 [ 4.098449] NET: Registered protocol family 17 [ 4.101560] mpls_gso: MPLS GSO support [ 4.109729] RAS: Correctable Errors collector initialized. [ 4.112324] AVX version of gcm_enc/dec engaged. [ 4.114092] AES CTR mode by8 optimization enabled [ 4.208485] sched_clock: Marking stable (4208453225, 0)->(5170704651, -962251426) [ 4.212273] registered taskstats version 1 [ 4.214334] Loading compiled-in X.509 certificates [ 4.216813] zswap: loaded using pool lzo/zbud [ 4.245830] Key type big_key registered [ 4.258141] Key type encrypted registered [ 4.259585] ima: No TPM chip found, activating TPM-bypass! [ 4.261903] ima: Allocated hash algorithm: sha1 [ 4.263664] ima: No architecture policies found [ 4.265269] evm: Initialising EVM extended attributes: [ 4.267240] evm: security.selinux [ 4.268528] evm: security.ima [ 4.269678] evm: security.capability [ 4.271067] evm: HMAC attrs: 0x1 [ 4.273880] rtc_cmos 00:05: setting system clock to 2025-11-17 00:43:21 UTC (1763340201) [ 4.280286] debug: unmapping init [mem 0xffffffff8f603000-0xffffffff8f7fffff] [ 4.283486] debug: unmapping init [mem 0xffffffff8e382000-0xffffffff8e658fff] [ 4.292097] Write protecting the kernel read-only data: 28672k [ 4.295137] debug: unmapping init [mem 0xffffffff8ca03000-0xffffffff8cbfffff] [ 4.298029] debug: unmapping init [mem 0xffffffff8d314000-0xffffffff8d3fffff] [ 4.334943] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.342547] systemd[1]: Detected virtualization kvm. [ 4.344110] systemd[1]: Detected architecture x86-64. [ 4.345629] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.367798] systemd[1]: No hostname configured. [ 4.369199] systemd[1]: Set hostname to . [ 4.370976] random: systemd: uninitialized urandom read (16 bytes read) [ 4.373046] systemd[1]: Initializing machine ID from random generator. [ 4.426460] random: ln: uninitialized urandom read (6 bytes read) [ 4.516659] random: systemd: uninitialized urandom read (16 bytes read) [ 4.519259] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.523423] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 4.529505] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. Starting Journal Service... Starting Apply Kernel Variables... [ OK ] Reached target Sockets. [ OK ] Reached target Timers. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 5.183267] device-mapper: uevent: version 1.0.3 [ 5.185291] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.858543] random: fast init done [ 5.871084] virtio_net virtio0 ens2: renamed from eth0 [ 5.921240] scsi host0: ata_piix [ 5.935709] scsi host1: ata_piix [ 5.937150] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.939163] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.320918] dracut-initqueue[586]: RTNETLINK answers: File exists [ 10.775907] random: crng init done [ 10.777329] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 11.194208] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.327652] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.583056] SELinux: Disabled at runtime. [ 12.648989] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.657362] systemd[1]: Detected virtualization kvm. [ 12.659309] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.191946] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.195363] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.199767] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.203256] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.206439] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.213948] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.221145] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Listening on udev Control Socket. Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target Paths. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.au[ 13.339555] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS tomount' for details. Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.659454] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.891033] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.948917] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.101469] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.110231] EDAC sbridge: Ver: 1.1.2 [ 15.600303] Key type dns_resolver registered [ 15.915326] NFS: Registering the id_resolver key type [ 15.917361] Key type id_resolver registered [ 15.919600] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg227-server login: [ 42.979398] libcfs: loading out-of-tree module taints kernel. [ 43.008068] Key type ._llcrypt registered [ 43.009506] Key type .llcrypt registered [ 43.145173] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 59.385371] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 61.131596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 61.149523] alg: No test for adler32 (adler32-zlib) [ 62.572534] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 63.701640] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 65.487181] Key type lgssc registered [ 67.313923] Lustre: Echo OBD driver; http://www.lustre.org/ [ 69.531790] hrtimer: interrupt took 3779708 ns [ 86.914813] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 87.771193] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 95.589829] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 102.551109] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 109.724331] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 124.895695] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 136.893742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 136.952674] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 136.973720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 138.201609] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 138.249416] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 138.365460] Lustre: lustre-MDT0000: new disk, initializing [ 138.465484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 138.486292] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 141.915341] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 154.167074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 154.252147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 154.315131] Lustre: 6508:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 154.354268] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 154.357503] Lustre: Skipped 1 previous similar message [ 154.420706] Lustre: lustre-MDT0001: new disk, initializing [ 154.475211] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 154.523834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 154.539339] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 158.380699] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 163.204996] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 172.696333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 172.760372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 172.926775] Lustre: lustre-OST0000: new disk, initializing [ 172.933236] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 172.996177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 177.813857] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 179.212408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 179.216850] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 179.302425] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 191.274870] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 191.384704] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 191.464482] Lustre: lustre-OST0001: new disk, initializing [ 191.476269] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 191.525765] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 196.340553] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 196.635940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 196.640472] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 196.711677] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 207.251723] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 211.069771] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 222.038214] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing check_logdir /tmp/testlogs/ [ 227.050497] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing yml_node [ 232.175154] Lustre: DEBUG MARKER: Client: 2.16.61.49 [ 234.635904] Lustre: DEBUG MARKER: MDS: 2.16.61.49 [ 237.464837] Lustre: DEBUG MARKER: OSS: 2.16.61.49 [ 239.053866] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Nov 16 19:47:14 EST 2025 [ 255.509790] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 257.248315] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 272.356060] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 272.384740] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 272.403220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 273.377913] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 273.378268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 273.408503] Lustre: Skipped 1 previous similar message [ 278.496188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 278.504666] Lustre: Skipped 4 previous similar messages [ 278.559245] LustreError: 12216:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 278.706651] Lustre: server umount lustre-MDT0000 complete [ 282.944833] LustreError: 7895:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763340480 with bad export cookie 12932955730545986412 [ 282.951041] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 282.958925] LustreError: 7895:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 283.616299] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 283.622902] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 283.638687] Lustre: Skipped 3 previous similar messages [ 283.649651] Lustre: Skipped 1 previous similar message [ 287.648129] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 287.651974] Lustre: Skipped 1 previous similar message [ 289.224339] LustreError: 12417:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 289.233923] LustreError: 12417:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 289.468405] Lustre: server umount lustre-MDT0001 complete [ 293.328077] LustreError: 12619:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 293.348764] LustreError: 12619:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 293.414843] Lustre: server umount lustre-OST0000 complete [ 296.735474] LustreError: 12819:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 296.742745] LustreError: 12819:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 296.952680] Lustre: server umount lustre-OST0001 complete [ 302.667827] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 311.706561] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 320.760708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 327.165442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 333.784970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 340.186974] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 346.992246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 347.074632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 347.240433] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 347.272091] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 347.348482] Lustre: lustre-MDT0000: new disk, initializing [ 347.443405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 347.466029] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 351.101202] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 360.741138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 360.801033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 360.964332] Lustre: 15008:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 361.026045] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 361.033386] Lustre: Skipped 1 previous similar message [ 361.108396] Lustre: lustre-MDT0001: new disk, initializing [ 361.223577] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 361.253747] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 361.270300] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 365.404617] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 374.411939] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 379.264428] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 387.509246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 387.566911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 387.914857] Lustre: lustre-OST0000: new disk, initializing [ 387.917350] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 388.069383] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 389.954600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 389.978141] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 390.211512] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 393.986707] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 401.221096] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 405.072411] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 405.255118] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 409.087593] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 409.232967] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 415.714529] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 415.722349] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 415.729485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 415.738716] Lustre: Skipped 1 previous similar message [ 417.087773] LustreError: 17945:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 417.098090] LustreError: 17945:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 417.242901] Lustre: server umount lustre-OST0000 complete [ 420.833473] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 420.846991] Lustre: Skipped 1 previous similar message [ 426.824596] LustreError: 18147:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 426.832920] LustreError: 18147:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 426.975666] Lustre: server umount lustre-MDT0000 complete [ 430.384868] LustreError: 15000:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763340627 with bad export cookie 12932955730545987945 [ 430.389160] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 430.411627] LustreError: 15000:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 430.825944] Lustre: server umount lustre-MDT0001 complete [ 434.568819] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 19:50:30 (1763340630) [ 436.975203] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 438.536856] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 19:50:34 (1763340634) [ 444.774361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 445.163715] LustreError: 19018:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 445.174648] LustreError: 19018:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 445.218069] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 448.535273] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 450.564769] LustreError: 19018:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 455.672251] LustreError: 19018:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 456.195070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 460.387744] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 465.744269] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 468.943453] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 474.484148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 480.256119] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 485.912490] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 506.852322] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 506.857781] LustreError: Skipped 1 previous similar message [ 506.860455] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 506.876414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 506.882020] Lustre: Skipped 5 previous similar messages [ 509.584188] LustreError: 21284:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 509.604665] LustreError: 21284:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 509.688472] Lustre: server umount lustre-OST0000 complete [ 512.992956] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 512.998709] LustreError: Skipped 1 previous similar message [ 527.328578] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 527.494413] Lustre: server umount lustre-MDT0000 complete [ 528.352880] LustreError: 19018:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 528.352880] LustreError: 19014:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 530.676555] LustreError: 18999:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763340727 with bad export cookie 12932955730545988827 [ 530.678678] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 530.683277] LustreError: 18999:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 531.026917] Lustre: server umount lustre-MDT0001 complete [ 539.955418] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 542.164379] Key type lgssc unregistered [ 542.423785] LNet: 22222:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 542.435220] LNetError: 22222:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 542.450529] LNet: Removed LNI 192.168.202.127@tcp [ 543.165311] Key type .llcrypt unregistered [ 543.168435] Key type ._llcrypt unregistered [ 563.101068] Key type ._llcrypt registered [ 563.102601] Key type .llcrypt registered [ 563.232035] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 564.221440] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 564.298757] alg: No test for adler32 (adler32-zlib) [ 565.415531] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 565.640276] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 567.312459] Key type lgssc registered [ 568.276966] Lustre: Echo OBD driver; http://www.lustre.org/ [ 576.007374] loop: module loaded [ 576.110612] loop0: detected capacity change from 0 to 204800000 [ 576.362363] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 576.377341] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 576.704950] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 576.774858] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 582.357332] loop0: detected capacity change from 0 to 204800000 [ 582.542929] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 582.557838] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 582.565079] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 582.699417] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 587.193776] loop0: detected capacity change from 0 to 204800000 [ 592.269772] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 592.324637] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 592.351716] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 593.763532] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 593.852428] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 594.055181] Lustre: lustre-MDT0000: new disk, initializing [ 594.244791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 594.292304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 598.758486] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 607.089298] loop1: detected capacity change from 0 to 204800000 [ 612.392638] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 612.462051] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 612.790945] Lustre: lustre-OST07c6: new disk, initializing [ 612.796350] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 612.802541] Lustre: Skipped 1 previous similar message [ 612.885374] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 618.967262] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 619.641030] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 619.655686] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 619.693876] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 623.304109] Lustre: 24382:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 623.321044] Lustre: 24382:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 623.324670] Lustre: 24382:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 623.328275] Lustre: 24382:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 623.375350] Lustre: 24382:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 623.388608] Lustre: 24382:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 640.658700] Lustre: 24380:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 640.664739] Lustre: 24380:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 640.669705] Lustre: 24380:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 640.673481] Lustre: 24380:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 640.679728] Lustre: 24380:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 640.684851] Lustre: 24380:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 644.892401] Lustre: 24381:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 644.895738] Lustre: 24381:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 1 previous similar message [ 644.898913] Lustre: 24381:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 644.904404] Lustre: 24381:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 1 previous similar message [ 644.908257] Lustre: 24381:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 644.913540] Lustre: 24381:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 1 previous similar message [ 644.918030] Lustre: 24381:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 644.922040] Lustre: 24381:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 1 previous similar message [ 644.925627] Lustre: 24381:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 644.930905] Lustre: 24381:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 1 previous similar message [ 644.934411] Lustre: 24381:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 644.938326] Lustre: 24381:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 648.122568] Lustre: 24381:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 648.139349] Lustre: 24381:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 1 previous similar message [ 648.153632] Lustre: 24381:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 648.166931] Lustre: 24381:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 1 previous similar message [ 648.171627] Lustre: 24381:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 648.175954] Lustre: 24381:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 1 previous similar message [ 648.180038] Lustre: 24381:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 648.185649] Lustre: 24381:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 1 previous similar message [ 648.189581] Lustre: 24381:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 648.196767] Lustre: 24381:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 1 previous similar message [ 648.201574] Lustre: 24381:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 648.205262] Lustre: 24381:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 652.562363] Lustre: 24381:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 652.567721] Lustre: 24381:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 1 previous similar message [ 652.575956] Lustre: 24381:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 652.584951] Lustre: 24381:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 1 previous similar message [ 652.594842] Lustre: 24381:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 652.604043] Lustre: 24381:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 1 previous similar message [ 652.608032] Lustre: 24381:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 652.612744] Lustre: 24381:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 1 previous similar message [ 652.617053] Lustre: 24381:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 652.624357] Lustre: 24381:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 1 previous similar message [ 652.634964] Lustre: 24381:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 652.642934] Lustre: 24381:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 663.507521] Lustre: 24382:0:(osd_internal.h:1472:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 663.532575] Lustre: 24382:0:(osd_internal.h:1472:osd_trans_exec_op()) Skipped 5 previous similar messages [ 663.540658] Lustre: 24382:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 663.546028] Lustre: 24382:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 663.552340] Lustre: 24382:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 663.560362] Lustre: 24382:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 663.572822] Lustre: 24382:0:(osd_handler.c:2099:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 663.575466] Lustre: 24382:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 663.582512] Lustre: 24382:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 663.592632] Lustre: 24382:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 663.596541] Lustre: 24382:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 663.600573] Lustre: 24382:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 711.649878] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 711.659881] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 711.673885] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 716.548322] LustreError: 26572:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 716.675262] Lustre: server umount lustre-OST07c6 complete [ 726.223657] LustreError: 27074:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 726.234760] LustreError: 27074:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 726.465552] Lustre: server umount lustre-MDT0000 complete [ 742.229490] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 744.755613] Key type lgssc unregistered [ 745.053643] LNet: 27908:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 745.064235] LNetError: 27908:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 746.087866] LNet: Removed LNI 192.168.202.127@tcp [ 746.813196] Key type .llcrypt unregistered [ 746.814975] Key type ._llcrypt unregistered [ 764.326114] Key type ._llcrypt registered [ 764.327882] Key type .llcrypt registered [ 764.452827] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 765.407926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 765.415121] alg: No test for adler32 (adler32-zlib) [ 766.518248] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 766.713661] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 768.415859] Key type lgssc registered [ 769.283123] Lustre: Echo OBD driver; http://www.lustre.org/ [ 776.899310] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 776.920087] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 778.437538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 782.214782] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 787.943026] LustreError: 29158:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 787.958497] LustreError: 29158:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 789.422419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 789.760029] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 793.676022] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 798.821935] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 802.376156] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 808.749471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 809.133080] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 814.316805] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 819.178589] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 823.541210] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 839.147713] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 839.150627] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 839.164704] Lustre: Skipped 1 previous similar message [ 839.166923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 839.180415] Lustre: Skipped 1 previous similar message [ 843.334142] LustreError: 31606:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 843.469298] Lustre: server umount lustre-OST0000 complete [ 849.377281] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 849.380177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 849.389076] Lustre: Skipped 1 previous similar message [ 853.178040] LustreError: 31808:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 853.188534] LustreError: 31808:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 853.252891] Lustre: server umount lustre-MDT0000 complete [ 854.496470] LustreError: 29159:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 854.503356] LustreError: 29159:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 856.137089] LustreError: 29139:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763341053 with bad export cookie 7160287812726262715 [ 856.144075] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 856.250773] LustreError: 32009:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 856.255678] LustreError: 32009:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 856.367973] Lustre: server umount lustre-MDT0001 complete [ 865.556955] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 868.077474] Key type lgssc unregistered [ 868.290796] LNet: 32542:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 868.301821] LNetError: 32542:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 868.322149] LNet: Removed LNI 192.168.202.127@tcp [ 869.113507] Key type .llcrypt unregistered [ 869.115411] Key type ._llcrypt unregistered [ 885.366605] Key type ._llcrypt registered [ 885.368799] Key type .llcrypt registered [ 885.465626] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 886.071376] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 886.122976] alg: No test for adler32 (adler32-zlib) [ 887.123763] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 887.387071] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 889.071221] Key type lgssc registered [ 890.035601] Lustre: Echo OBD driver; http://www.lustre.org/ [ 896.533249] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 896.549552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 897.917336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 900.835944] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 908.260054] LustreError: 33795:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 908.286562] LustreError: 33795:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 908.695800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 909.014351] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 912.285638] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 917.272836] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 920.524945] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 926.601985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 926.892579] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 931.710986] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 937.034170] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 943.290312] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 958.947871] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 958.961470] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 958.962652] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 958.967315] Lustre: Skipped 1 previous similar message [ 961.851124] LustreError: 36066:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 961.966820] Lustre: server umount lustre-OST0000 complete [ 965.600200] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 965.608839] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 965.615690] Lustre: Skipped 1 previous similar message [ 965.621483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 969.184546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 969.188908] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 971.118203] LustreError: 36268:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 971.120940] LustreError: 36268:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 971.208951] Lustre: server umount lustre-MDT0000 complete [ 974.046325] LustreError: 33777:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763341171 with bad export cookie 6199470349750084439 [ 974.053614] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 974.152711] LustreError: 36469:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 974.156168] LustreError: 36469:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 974.302883] Lustre: server umount lustre-MDT0001 complete [ 981.584124] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 983.349437] Key type lgssc unregistered [ 983.557423] LNet: 37001:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 983.560747] LNetError: 37001:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 983.575524] LNet: Removed LNI 192.168.202.127@tcp [ 984.023222] Key type .llcrypt unregistered [ 984.025305] Key type ._llcrypt unregistered [ 1002.683806] Key type ._llcrypt registered [ 1002.685884] Key type .llcrypt registered [ 1002.795896] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 1003.695163] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1003.729218] alg: No test for adler32 (adler32-zlib) [ 1004.770286] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 1004.935091] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 1006.583220] Key type lgssc registered [ 1007.681727] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1014.541627] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1014.560608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1015.885830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1019.489272] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1026.018852] LustreError: 38379:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1026.036122] LustreError: 38379:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1027.788462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1028.133716] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1031.529048] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1037.183778] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1040.960608] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1046.579900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1046.789241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1050.662525] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1054.325619] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1061.110192] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1071.308868] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.27@tcp (stopping) [ 1071.590825] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1071.597788] Lustre: Skipped 2 previous similar messages [ 1072.980575] LustreError: 40658:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1073.215768] Lustre: server umount lustre-MDT0000 complete [ 1076.400613] LustreError: 38375:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1077.098345] LustreError: 38360:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763341274 with bad export cookie 7670549455353292498 [ 1077.110669] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1077.118230] LustreError: 38360:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1081.514050] LustreError: 38374:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1081.527467] LustreError: 38374:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1081.540561] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.27@tcp (stopping) [ 1081.543946] Lustre: Skipped 3 previous similar messages [ 1081.825082] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1082.848897] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1082.856942] Lustre: Skipped 1 previous similar message [ 1083.296866] LustreError: 40859:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1083.299723] LustreError: 40859:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1083.502257] Lustre: server umount lustre-MDT0001 complete [ 1103.622449] LustreError: 41084:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1103.625590] LustreError: 41084:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1103.742066] Lustre: server umount lustre-OST0000 complete [ 1119.071170] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 1121.495224] Key type lgssc unregistered [ 1121.740081] LNet: 41921:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1121.749162] LNetError: 41921:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1121.773068] LNet: Removed LNI 192.168.202.127@tcp [ 1122.616450] Key type .llcrypt unregistered [ 1122.629391] Key type ._llcrypt unregistered [ 1142.159223] Key type ._llcrypt registered [ 1142.161236] Key type .llcrypt registered [ 1142.258753] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 1143.029099] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1143.098081] alg: No test for adler32 (adler32-zlib) [ 1144.132829] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 1144.338570] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 1146.015210] Key type lgssc registered [ 1147.113757] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1153.793540] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1153.826705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1155.183341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1158.650894] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1164.779169] LustreError: 43299:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1164.795242] LustreError: 43299:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1166.289465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1166.550338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1169.469331] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1173.496134] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1176.944414] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1183.964787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1184.250192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1188.932365] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1193.853198] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1200.635145] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1210.069212] Lustre: lustre-OST0000: Not available for connect from 192.168.202.27@tcp (stopping) [ 1210.848138] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1210.855663] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1210.871176] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1210.878042] Lustre: Skipped 1 previous similar message [ 1210.897929] Lustre: Skipped 1 previous similar message [ 1215.163879] Lustre: lustre-OST0000: Not available for connect from 192.168.202.27@tcp (stopping) [ 1215.327120] LustreError: 45573:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1215.526335] Lustre: server umount lustre-OST0000 complete [ 1233.376228] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1233.389229] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1233.410640] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1236.449415] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1238.655525] LustreError: 45948:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1238.658030] LustreError: 45948:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1238.815789] Lustre: server umount lustre-MDT0000 complete [ 1241.571577] LustreError: 43299:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1241.595802] LustreError: 43299:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1241.996663] LustreError: 43280:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763341439 with bad export cookie 12149188958108077454 [ 1241.998356] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1242.001916] LustreError: 43280:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1248.253697] LustreError: 46150:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1248.259686] LustreError: 46150:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1248.412069] Lustre: server umount lustre-MDT0001 complete [ 1256.480957] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 1258.158707] Key type lgssc unregistered [ 1258.326930] LNet: 46682:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1258.334497] LNetError: 46682:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1258.343961] LNet: Removed LNI 192.168.202.127@tcp [ 1258.860394] Key type .llcrypt unregistered [ 1258.862358] Key type ._llcrypt unregistered [ 1279.647404] Key type ._llcrypt registered [ 1279.649464] Key type .llcrypt registered [ 1279.715916] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 1280.570117] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1280.600084] alg: No test for adler32 (adler32-zlib) [ 1281.603945] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 1281.882907] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 1283.543182] Key type lgssc registered [ 1284.414904] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1290.528329] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1290.572692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1292.058827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1294.674554] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1300.921206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1301.144418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1303.414732] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1307.804163] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1310.997449] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1316.736510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1317.048143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1318.057145] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1321.545977] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1326.288193] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1330.525121] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1332.628046] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1344.414750] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2025-11-1620h05m39s [ 1345.673549] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1347.553957] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1347.561784] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1347.573478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1348.282390] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.27@tcp (stopping) [ 1348.643443] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1348.650896] Lustre: Skipped 1 previous similar message [ 1349.352888] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.27@tcp (stopping) [ 1349.356269] Lustre: Skipped 3 previous similar messages [ 1351.722299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1351.732905] Lustre: Skipped 4 previous similar messages [ 1353.316651] LustreError: 50642:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1353.404457] Lustre: server umount lustre-MDT0000 complete [ 1353.762343] LustreError: 50644:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1353.768727] LustreError: 50644:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1354.409616] LustreError: 50644:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1355.827477] LustreError: 50644:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1355.855616] LustreError: 50644:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1356.089042] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1357.858332] LustreError: 48075:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1357.876043] LustreError: 48075:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1361.163637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1361.247777] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1361.420258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1361.461413] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1362.469544] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1363.885360] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1366.179315] Lustre: DEBUG MARKER: Wait for df (36040) ... [ 1367.409459] Lustre: DEBUG MARKER: done [ 1371.616188] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1371.626205] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1371.637493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1371.640303] Lustre: Skipped 1 previous similar message [ 1372.643085] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1375.879898] LustreError: 51699:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1375.889984] LustreError: 51699:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1375.982039] Lustre: server umount lustre-OST0000 complete [ 1382.881052] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1382.885341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1382.912926] Lustre: Skipped 2 previous similar messages [ 1382.913862] Lustre: Skipped 2 previous similar messages [ 1385.048619] LustreError: 51900:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1385.052859] LustreError: 51900:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1385.162531] Lustre: server umount lustre-MDT0000 complete [ 1388.171366] LustreError: 50374:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763341585 with bad export cookie 15047784069388207903 [ 1388.180502] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1388.181173] LustreError: 50374:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1388.298695] LustreError: 52102:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1388.310429] LustreError: 52102:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1388.474172] Lustre: server umount lustre-MDT0001 complete [ 1397.554759] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 1399.924435] Key type lgssc unregistered [ 1400.173185] LNet: 52632:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1400.184376] LNetError: 52632:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1400.199030] LNet: Removed LNI 192.168.202.127@tcp [ 1400.865911] Key type .llcrypt unregistered [ 1400.868859] Key type ._llcrypt unregistered [ 1430.629863] Key type ._llcrypt registered [ 1430.631437] Key type .llcrypt registered [ 1430.713962] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 1444.332355] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 1445.209117] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1445.292367] alg: No test for adler32 (adler32-zlib) [ 1446.321438] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 1446.562039] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 1448.223183] Key type lgssc registered [ 1449.229607] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1457.855337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1464.272642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1470.248086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1476.458060] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1483.010468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1483.076738] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1483.089242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1484.386447] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1484.436473] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1484.501332] Lustre: lustre-MDT0000: new disk, initializing [ 1484.591440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1484.613922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1488.403797] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1499.434402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1499.518508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1499.722271] Lustre: 56580:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1499.816842] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1499.823818] Lustre: Skipped 1 previous similar message [ 1499.945059] Lustre: lustre-MDT0001: new disk, initializing [ 1500.075394] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1500.139170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1500.151556] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1504.030584] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1511.337786] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1515.958560] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1523.334901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1523.458353] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1523.808710] Lustre: lustre-OST0000: new disk, initializing [ 1523.821452] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1523.909292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1528.624418] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1531.962480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1531.972368] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1532.027151] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1535.053865] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1538.701507] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1538.925694] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1542.829252] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1543.006444] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1547.236320] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1547.242219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1547.255866] Lustre: Skipped 1 previous similar message [ 1547.265359] Lustre: Skipped 1 previous similar message [ 1552.355476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1552.358184] Lustre: Skipped 1 previous similar message [ 1557.476404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1557.486927] Lustre: Skipped 1 previous similar message [ 1559.007929] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1559.035808] LustreError: 59529:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1559.155562] Lustre: server umount lustre-OST0000 complete [ 1562.592281] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1562.595453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1562.599819] Lustre: Skipped 1 previous similar message [ 1567.713817] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1567.719741] Lustre: Skipped 2 previous similar messages [ 1568.387488] LustreError: 59734:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1568.394233] LustreError: 59734:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1568.519573] Lustre: server umount lustre-MDT0000 complete [ 1571.418814] LustreError: 56573:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763341768 with bad export cookie 16059194297813766855 [ 1571.419617] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1571.428310] LustreError: 56573:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1571.560243] LustreError: 59936:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1571.565628] LustreError: 59936:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1571.681858] Lustre: server umount lustre-MDT0001 complete [ 1578.754956] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 20:09:34 (1763341774) [ 1584.035302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1584.518899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1587.977584] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1589.729096] LustreError: 60609:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1589.737172] LustreError: 60609:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1593.846725] LustreError: 60609:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1595.554161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1595.956068] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1599.661969] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1605.604956] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1609.690344] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1616.029664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1616.280279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1620.833383] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1625.590599] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1635.514577] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2025-11-1620h10m31s [ 1637.097907] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1643.397639] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1645.810806] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1645.859239] LustreError: 62584:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1666.254712] Lustre: lustre-MDT0000: Client c2782ca6-cc07-49e4-bd75-0ad2852dcdf0 (at 192.168.202.27@tcp) reconnecting [ 1685.863139] LustreError: 62584:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1687.803271] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1689.901832] Lustre: DEBUG MARKER: done [ 1696.224271] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1696.226380] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1696.237862] Lustre: Skipped 1 previous similar message [ 1696.241246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1697.070871] LustreError: 63612:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1697.074049] LustreError: 63612:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1697.243424] Lustre: server umount lustre-OST0000 complete [ 1701.346757] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1701.361509] Lustre: Skipped 1 previous similar message [ 1706.870543] LustreError: 63813:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1706.876942] LustreError: 63813:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1706.958575] Lustre: server umount lustre-MDT0000 complete [ 1707.490638] LustreError: 60603:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1707.516650] LustreError: 60603:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1710.539366] LustreError: 60588:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763341907 with bad export cookie 16059194297813767737 [ 1710.540169] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1710.556112] LustreError: 60588:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1710.918381] Lustre: server umount lustre-MDT0001 complete [ 1720.490963] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 1723.300876] Key type lgssc unregistered [ 1723.631525] LNet: 64546:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1723.647050] LNetError: 64546:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1723.670347] LNet: Removed LNI 192.168.202.127@tcp [ 1724.575975] Key type .llcrypt unregistered [ 1724.577412] Key type ._llcrypt unregistered [ 1756.178514] Key type ._llcrypt registered [ 1756.180323] Key type .llcrypt registered [ 1756.323981] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 1771.254766] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 1772.547872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1772.635615] alg: No test for adler32 (adler32-zlib) [ 1773.823318] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 1774.088037] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 1775.807926] Key type lgssc registered [ 1777.532503] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1787.738249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1794.724343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1800.495080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1806.527061] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1811.939756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1811.981603] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1811.992903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1813.164131] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1813.189909] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1813.238985] Lustre: lustre-MDT0000: new disk, initializing [ 1813.326126] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1813.343143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1816.284609] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1825.739129] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1825.787293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1825.893524] Lustre: 68498:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1825.946947] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1825.950833] Lustre: Skipped 1 previous similar message [ 1826.022894] Lustre: lustre-MDT0001: new disk, initializing [ 1826.164507] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1826.199537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1826.213252] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1829.854875] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1837.301644] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1840.820222] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1847.726492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1847.814960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1848.073555] Lustre: lustre-OST0000: new disk, initializing [ 1848.083422] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1848.206169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1853.524128] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1857.162394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1857.196510] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1857.300271] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1861.626767] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1865.941374] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1866.208370] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1869.852752] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1870.084156] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1872.867158] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1872.870436] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1872.880324] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1877.989081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1877.998151] Lustre: Skipped 1 previous similar message [ 1878.143177] LustreError: 71439:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1878.313559] Lustre: server umount lustre-OST0000 complete [ 1882.599105] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1882.602833] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1882.625917] Lustre: Skipped 1 previous similar message [ 1882.636116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1882.643805] Lustre: Skipped 1 previous similar message [ 1888.063589] LustreError: 71641:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1888.078384] LustreError: 71641:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1888.222475] Lustre: server umount lustre-MDT0000 complete [ 1891.657856] LustreError: 69279:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342088 with bad export cookie 12840633261523643091 [ 1891.670381] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1891.843426] LustreError: 71843:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1891.846411] LustreError: 71843:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1892.011979] Lustre: server umount lustre-MDT0001 complete [ 1899.284806] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 20:14:55 (1763342095) [ 1900.980220] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1902.778618] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 20:14:58 (1763342098) [ 1913.148279] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 1917.211416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1917.672543] LustreError: 72890:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1917.690525] LustreError: 72890:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1917.733735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1918.746663] Lustre: Failing over lustre-MDT0000 [ 1918.758051] LustreError: 72918:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1918.771477] LustreError: 72918:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1918.787798] LustreError: 72918:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 2, retries 0, failed: rc = -5 [ 1918.926961] LustreError: 72970:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1918.933232] LustreError: 72970:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1919.114050] Lustre: server umount lustre-MDT0000 complete [ 1925.164213] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 20:15:20 (1763342120) [ 1931.080649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1931.487598] LustreError: 73537:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1931.588334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1935.171946] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1936.868711] LustreError: 73537:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1940.973328] LustreError: 73537:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1943.998108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1944.315434] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1947.779349] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1953.228241] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1956.630736] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1963.594945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1964.023013] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1969.987895] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 1975.822698] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1981.311522] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1990.629370] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1990.632431] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1990.648381] Lustre: Skipped 1 previous similar message [ 1990.656125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1990.661442] Lustre: Skipped 2 previous similar messages [ 1996.773530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1996.776592] Lustre: Skipped 4 previous similar messages [ 2001.375093] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2001.493627] LustreError: 75881:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2001.497322] LustreError: 75881:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2001.629839] Lustre: server umount lustre-MDT0000 complete [ 2001.890180] LustreError: 75739:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2001.920632] LustreError: 75739:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2005.146147] LustreError: 73519:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342202 with bad export cookie 12840633261523644246 [ 2005.152252] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2005.157774] LustreError: 73519:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2005.708802] Lustre: server umount lustre-MDT0001 complete [ 2009.651809] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2014.221791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2021.817160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2023.392065] Lustre: 66710:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763342204/real 1763342204] req@0000000089ba132e x1848998070656384/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763342220 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2023.430359] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2023.439230] Lustre: Skipped 2 previous similar messages [ 2026.528046] Lustre: 66710:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763342207/real 1763342207] req@0000000067759f73 x1848998070656512/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763342223 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2028.511243] Lustre: 66708:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763342209/real 1763342209] req@00000000c5f4c3d4 x1848998070656640/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763342225 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2030.623674] LustreError: 66706:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000e101b21b x1848998070657280/t0(0) o250->MGC192.168.202.127@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2030.882576] LustreError: 75146:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2030.952329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2031.051630] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2031.973182] Lustre: 66708:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763342212/real 1763342212] req@0000000044d1922e x1848998070656896/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763342228 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2031.981679] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2034.568462] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 2036.063175] Lustre: 66710:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763342217/real 1763342217] req@00000000e3fcd9cc x1848998070657024/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763342233 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2043.275132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2047.073797] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 2049.003299] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2052.003853] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2055.319704] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2061.419945] LustreError: 78343:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2061.435719] LustreError: 78343:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2061.579975] Lustre: server umount lustre-MDT0000 complete [ 2064.354649] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2064.358022] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2064.362742] Lustre: Skipped 1 previous similar message [ 2064.364724] LustreError: 78160:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2064.382115] LustreError: 78160:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 2064.852919] LustreError: 73518:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342262 with bad export cookie 12840633261523646150 [ 2064.860585] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2064.869055] LustreError: 73518:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2065.160957] Lustre: server umount lustre-MDT0001 complete [ 2071.039238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2080.211947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2084.837716] Lustre: 66710:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763342266/real 1763342266] req@000000007a306619 x1848998070680832/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1763342282 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2084.857169] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2084.864600] Lustre: Skipped 1 previous similar message [ 2090.017792] LustreError: 66706:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d70b77ad x1848998070681856/t0(0) o250->MGC192.168.202.127@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2090.476339] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2094.442119] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 2100.711716] LustreError: 79400:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2100.721554] LustreError: 79400:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2102.546519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2102.965130] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2102.967802] Lustre: Skipped 2 previous similar messages [ 2103.984860] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2107.380787] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 2113.660865] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2117.464101] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2123.907121] LustreError: 80802:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2123.911088] LustreError: 80802:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2123.981535] Lustre: server umount lustre-MDT0000 complete [ 2124.769466] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2127.832267] LustreError: 76087:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342325 with bad export cookie 12840633261523647522 [ 2127.833030] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2127.841227] LustreError: 76087:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2128.197182] Lustre: server umount lustre-MDT0001 complete [ 2135.725585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2139.624644] Lustre: DEBUG MARKER: files compared the same [ 2152.528756] Lustre: server umount lustre-OST0000 complete [ 2166.055890] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 2168.281909] Key type lgssc unregistered [ 2168.514230] LNet: 82436:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2168.519512] LNetError: 82436:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2168.535851] LNet: Removed LNI 192.168.202.127@tcp [ 2169.232478] Key type .llcrypt unregistered [ 2169.245331] Key type ._llcrypt unregistered [ 2187.957670] Key type ._llcrypt registered [ 2187.958816] Key type .llcrypt registered [ 2188.056982] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 2189.120019] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2189.126484] alg: No test for adler32 (adler32-zlib) [ 2190.118444] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 2190.343641] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 2192.007845] Key type lgssc registered [ 2193.060806] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2201.035696] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2201.044485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2202.371552] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2206.041422] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug +malloc all [ 2211.816637] LustreError: 83812:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2211.833909] LustreError: 83812:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2215.582458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2215.951679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2220.396901] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug +malloc all [ 2225.846454] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2230.737955] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2239.339226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2239.706666] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2246.419885] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug +malloc all [ 2246.898769] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2251.788117] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2262.496265] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2262.535656] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2262.574433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2262.581219] Lustre: Skipped 1 previous similar message [ 2264.456095] LustreError: 86061:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2264.617097] Lustre: server umount lustre-OST0000 complete [ 2272.229332] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2272.236110] LustreError: Skipped 1 previous similar message [ 2272.243244] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2272.252253] Lustre: Skipped 1 previous similar message [ 2272.255080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2275.124603] LustreError: 86262:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2275.137515] LustreError: 86262:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2275.380233] Lustre: server umount lustre-MDT0000 complete [ 2277.855566] LustreError: 84810:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2277.870942] LustreError: 84810:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2279.120222] LustreError: 83792:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342476 with bad export cookie 1188370536299336312 [ 2279.126953] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2279.137721] LustreError: 83792:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2279.293215] LustreError: 86464:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2279.296241] LustreError: 86464:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2279.476340] Lustre: server umount lustre-MDT0001 complete [ 2292.081318] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 2295.374060] Key type lgssc unregistered [ 2295.881715] LNet: 86997:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2295.897640] LNetError: 86997:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2295.932365] LNet: Removed LNI 192.168.202.127@tcp [ 2297.093813] Key type .llcrypt unregistered [ 2297.095432] Key type ._llcrypt unregistered [ 2322.585351] Key type ._llcrypt registered [ 2322.586700] Key type .llcrypt registered [ 2322.694817] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 2323.500171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2323.508041] alg: No test for adler32 (adler32-zlib) [ 2324.618257] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 2324.831612] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 2326.519557] Key type lgssc registered [ 2327.496316] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2335.258662] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2335.270205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2350.751649] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2355.871191] LustreError: 88368:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.127@tcp: failed processing log, type 4: rc = -110 [ 2382.559682] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2388.300754] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 2393.579464] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2400.822607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2400.957843] Lustre: *** cfs_fail_loc=706, val=0*** [ 2401.273045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2401.371928] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2405.116588] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 2406.380537] LustreError: 88395:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2406.422879] LustreError: 88395:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2411.491131] LustreError: 88395:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2413.731626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2414.055215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2417.789674] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 2423.812734] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2427.450511] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2431.973623] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2431.976705] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2431.993647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2432.489891] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2432.523939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2432.526712] Lustre: Skipped 1 previous similar message [ 2435.713876] LustreError: 90694:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 2435.961644] Lustre: server umount lustre-OST0000 complete [ 2439.651214] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2439.654767] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2439.662419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2442.720588] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2442.722561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2442.733392] Lustre: Skipped 1 previous similar message [ 2445.850131] LustreError: 90895:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2445.865702] LustreError: 90895:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2446.048457] Lustre: server umount lustre-MDT0000 complete [ 2447.842694] LustreError: 89341:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2447.862737] LustreError: 89341:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2450.009790] LustreError: 88377:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763342647 with bad export cookie 6490272521160063466 [ 2450.012295] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2450.026758] LustreError: 88377:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2450.176231] LustreError: 91097:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2450.179381] LustreError: 91097:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2450.384692] Lustre: server umount lustre-MDT0001 complete [ 2460.804192] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 2463.286807] Key type lgssc unregistered [ 2463.499697] LNet: 91628:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2463.511479] LNetError: 91628:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2464.552318] LNet: Removed LNI 192.168.202.127@tcp [ 2465.206765] Key type .llcrypt unregistered [ 2465.210787] Key type ._llcrypt unregistered [ 2488.827430] Key type ._llcrypt registered [ 2488.832517] Key type .llcrypt registered [ 2488.914588] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 2489.875718] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2490.016788] alg: No test for adler32 (adler32-zlib) [ 2491.121646] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 2491.342987] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 2493.015300] Key type lgssc registered [ 2494.119860] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2501.701593] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2501.734442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2506.812354] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 2515.375837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2515.879500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2520.924283] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 2521.056822] LustreError: 93700:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2521.088971] LustreError: 93700:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2526.179698] LustreError: 93705:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2526.203470] LustreError: 93705:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2530.962847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2531.301130] LustreError: 93705:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2531.309577] LustreError: 93705:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2531.339190] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2536.423913] LustreError: 93705:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2537.320136] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 2541.538774] LustreError: 93704:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2541.575678] LustreError: 93704:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2545.654206] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2545.664195] Lustre: MGS: Client 5d6e8eac-833f-4984-8069-01d89852d62f (at 0@lo) reconnecting [ 2545.907894] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2546.030340] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2549.363959] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 2566.630439] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2566.633899] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2566.638738] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2566.659059] Lustre: Skipped 1 previous similar message [ 2568.634188] LustreError: 95664:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 2568.795483] Lustre: server umount lustre-OST0000 complete [ 2576.865229] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2576.899665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2576.905707] Lustre: Skipped 2 previous similar messages [ 2578.404425] LustreError: 95867:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2578.407014] LustreError: 95867:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2578.437837] Lustre: server umount lustre-MDT0000 complete [ 2581.989808] LustreError: 95559:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2582.004298] LustreError: 95559:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2582.350877] LustreError: 96069:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2582.354691] LustreError: 96069:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2582.532242] Lustre: server umount lustre-MDT0001 complete [ 2585.807098] LustreError: 96270:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2585.815123] LustreError: 96270:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2585.890460] Lustre: server umount MGS complete [ 2597.684230] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 20:26:33 (1763342793) [ 2627.328118] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 2638.279409] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 2647.924547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2654.114936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2659.846513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2666.047710] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2673.038451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2673.087249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2676.517263] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 2685.859917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2685.943637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2686.102591] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2686.154451] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2686.255842] Lustre: lustre-MDT0001: new disk, initializing [ 2686.402472] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2686.442299] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2686.460876] Lustre: 100789:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2686.469212] LustreError: 100789:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2690.837293] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 2701.676894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2701.762307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2701.991332] Lustre: lustre-OST0000: new disk, initializing [ 2701.993665] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2702.080713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2707.712819] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 2711.075958] Lustre: 101981:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000798d80cd x1848998822452096/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1763342924 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2715.352949] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2722.203632] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2722.221082] Lustre: MGS: Client 2a11a285-24d7-41ca-8fa7-a9a704ca4937 (at 0@lo) reconnecting [ 2722.289570] Lustre: 100072:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2722.309623] LustreError: 100072:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 2722.381727] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2722.446769] Lustre: lustre-MDT0000: new disk, initializing [ 2722.583719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2722.608706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2722.710786] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2723.682024] Lustre: 102501:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@0000000086a2c042 x1848998822458240/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1763342936 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2726.789852] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 2731.508187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2731.518617] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2732.843512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 2732.860763] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 2732.918593] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2758.625355] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2758.650491] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2758.687797] Lustre: Skipped 1 previous similar message [ 2758.691631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2764.279152] LustreError: 103022:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 2764.290585] LustreError: 103022:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2764.425446] Lustre: server umount lustre-OST0000 complete [ 2770.406397] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2770.412405] Lustre: Skipped 1 previous similar message [ 2770.417598] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2770.418925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2770.418931] Lustre: Skipped 2 previous similar messages [ 2774.312430] LustreError: 103223:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 2774.324564] LustreError: 103223:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2774.349399] Lustre: server umount lustre-MDT0000 complete [ 2775.537656] LustreError: 100765:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2775.547906] LustreError: 100765:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2784.781345] Lustre: server umount lustre-MDT0001 complete [ 2789.505947] Lustre: server umount MGS complete [ 2802.431947] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 20:29:57 (1763342997) [ 2820.053416] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 2823.197818] Key type lgssc unregistered [ 2823.441256] LNet: 104970:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2823.459367] LNetError: 104970:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2823.495267] LNet: Removed LNI 192.168.202.127@tcp [ 2824.355989] Key type .llcrypt unregistered [ 2824.357673] Key type ._llcrypt unregistered [ 2840.695757] Key type ._llcrypt registered [ 2840.699375] Key type .llcrypt registered [ 2840.798896] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 2841.921748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2842.095384] alg: No test for adler32 (adler32-zlib) [ 2843.284927] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 2843.586102] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 2845.304508] Key type lgssc registered [ 2846.640700] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2855.087306] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2855.102512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2856.730274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2861.125509] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 2866.163883] LustreError: 106170:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2866.185123] LustreError: 106170:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2870.415174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2870.846522] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2875.748170] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 2881.799147] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2886.163959] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2893.215167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2893.626107] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2899.713872] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 2901.816674] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 2904.507907] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2911.204058] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2911.217426] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2911.241229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2912.228038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2912.231021] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2912.251439] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2917.053680] LustreError: 108417:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2917.226517] Lustre: server umount lustre-OST0000 complete [ 2921.963652] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2921.966651] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2921.993566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2922.002891] Lustre: Skipped 1 previous similar message [ 2927.256814] LustreError: 108619:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2927.259784] LustreError: 108619:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2927.351605] Lustre: server umount lustre-MDT0000 complete [ 2927.586819] LustreError: 106171:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2930.953543] LustreError: 106151:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343128 with bad export cookie 17149114158605506030 [ 2930.962324] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2930.968394] LustreError: 106151:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2931.109020] LustreError: 108821:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2931.130278] LustreError: 108821:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2931.378908] Lustre: server umount lustre-MDT0001 complete [ 2939.898591] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 2942.481041] Key type lgssc unregistered [ 2942.749239] LNet: 109352:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2942.765256] LNetError: 109352:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2942.780061] LNet: Removed LNI 192.168.202.127@tcp [ 2943.581559] Key type .llcrypt unregistered [ 2943.583289] Key type ._llcrypt unregistered [ 2958.461367] Key type ._llcrypt registered [ 2958.462991] Key type .llcrypt registered [ 2958.626918] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 2959.808378] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2959.849192] alg: No test for adler32 (adler32-zlib) [ 2960.902608] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 2961.102689] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 2962.751254] Key type lgssc registered [ 2963.832457] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2969.414897] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2969.423208] LustreError: 110412:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 2969.457565] LustreError: 110417:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 2969.461961] LustreError: 110412:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4970 [ 2969.466580] LustreError: 110412:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2969.480493] LustreError: 110412:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2969.494801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2971.050939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2978.225225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2978.447515] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2982.058251] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 2986.333560] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing lsmod [ 2989.194285] LustreError: 111578:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 2989.269477] LustreError: 111594:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 2989.278927] LustreError: 111578:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4915 [ 2989.282697] LustreError: 111578:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2989.293722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2989.298158] LustreError: 111578:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2989.806352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2994.145694] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2994.148164] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2994.158387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2995.172043] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2995.174319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2995.186145] Lustre: Skipped 1 previous similar message [ 2995.196972] Lustre: Skipped 2 previous similar messages [ 2998.627807] LustreError: 111779:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2998.746173] Lustre: server umount lustre-MDT0000 complete [ 3000.297894] LustreError: 110450:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3000.321343] LustreError: 110450:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3002.259332] LustreError: 110895:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343199 with bad export cookie 16417213260681846169 [ 3002.266616] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3002.276396] LustreError: 110895:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3002.428814] LustreError: 111985:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3002.431184] LustreError: 111985:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3002.610190] Lustre: server umount lustre-MDT0001 complete [ 3016.779962] LustreError: 112186:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3016.783335] LustreError: 112186:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3017.026343] Lustre: server umount lustre-OST0000 complete [ 3025.335429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3025.718993] LustreError: 112663:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3025.785665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3029.913925] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3031.040642] LustreError: 112663:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3035.106392] LustreError: 112663:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3039.896755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3040.229610] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3040.442413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3044.567480] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3050.273220] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3054.116415] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3060.756865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3061.024487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3066.612345] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3070.273458] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 3072.072860] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3080.674665] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3080.677294] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3080.681081] Lustre: Skipped 1 previous similar message [ 3080.684196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3080.686065] Lustre: Skipped 1 previous similar message [ 3083.769626] LustreError: 114908:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3083.772192] LustreError: 114908:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3083.865047] Lustre: server umount lustre-OST0000 complete [ 3090.913949] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3090.921467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3090.924249] Lustre: Skipped 1 previous similar message [ 3093.371478] LustreError: 115113:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3093.377660] LustreError: 115113:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3093.496150] Lustre: server umount lustre-MDT0000 complete [ 3096.032505] LustreError: 113679:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3096.049818] LustreError: 113679:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3097.072333] LustreError: 112644:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343294 with bad export cookie 16417213260681846862 [ 3097.084045] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3097.096850] LustreError: 112644:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3097.508848] Lustre: server umount lustre-MDT0001 complete [ 3107.896727] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 3110.302170] Key type lgssc unregistered [ 3110.558676] LNet: 115847:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3110.567814] LNetError: 115847:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3110.596585] LNet: Removed LNI 192.168.202.127@tcp [ 3111.312816] Key type .llcrypt unregistered [ 3111.315700] Key type ._llcrypt unregistered [ 3133.559158] Key type ._llcrypt registered [ 3133.563943] Key type .llcrypt registered [ 3133.695803] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 3134.946064] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3134.979714] alg: No test for adler32 (adler32-zlib) [ 3136.097782] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 3136.294881] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 3137.959218] Key type lgssc registered [ 3139.042817] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3145.680496] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3145.692773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3147.090918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3150.577259] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3157.474277] LustreError: 117245:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3157.492333] LustreError: 117245:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3158.736519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3158.996456] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3162.889575] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3167.590233] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3171.375634] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3178.814598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3179.161206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3184.897025] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3190.353927] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3192.529042] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 3200.136967] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3202.339560] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3207.416114] LustreError: 119590:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3207.559893] Lustre: server umount lustre-OST0000 complete [ 3213.291935] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3213.299462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3213.309418] Lustre: Skipped 1 previous similar message [ 3213.324450] Lustre: Skipped 1 previous similar message [ 3218.400438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3218.408610] Lustre: Skipped 1 previous similar message [ 3221.480559] LustreError: 119795:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3221.487183] LustreError: 119795:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3221.570247] Lustre: server umount lustre-MDT0000 complete [ 3223.523151] LustreError: 117245:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3223.540645] LustreError: 117245:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3224.544140] Lustre: 116557:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763343405/real 1763343405] req@000000008675d925 x1848999498775552/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1763343421 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3224.562619] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3224.579121] LustreError: 117245:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3224.595404] LustreError: 117245:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3224.774952] LustreError: 117227:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343421 with bad export cookie 1301733133243252821 [ 3224.777237] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3224.787499] LustreError: 117227:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3224.887884] LustreError: 119997:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3224.890127] LustreError: 119997:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3225.010288] Lustre: server umount lustre-MDT0001 complete [ 3234.336263] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 3236.753856] Key type lgssc unregistered [ 3236.980447] LNet: 120528:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3236.985792] LNetError: 120528:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3238.057635] LNet: Removed LNI 192.168.202.127@tcp [ 3238.768451] Key type .llcrypt unregistered [ 3238.771620] Key type ._llcrypt unregistered [ 3253.052420] Key type ._llcrypt registered [ 3253.055661] Key type .llcrypt registered [ 3253.173539] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 3254.023882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3254.031971] alg: No test for adler32 (adler32-zlib) [ 3255.072952] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 3255.273272] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 3256.935178] Key type lgssc registered [ 3257.989391] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3265.034743] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3265.058981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3266.502789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3270.205286] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3276.788211] LustreError: 121731:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3276.801287] LustreError: 121731:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3278.656085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3278.945428] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3282.357911] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3287.418619] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3290.616678] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3296.753325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3297.092161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3301.297916] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3305.727765] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3312.495712] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3322.850711] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3322.857286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3322.857449] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3322.874679] Lustre: Skipped 1 previous similar message [ 3327.973787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3327.980114] Lustre: Skipped 2 previous similar messages [ 3328.193120] LustreError: 123999:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3328.300565] Lustre: server umount lustre-OST0000 complete [ 3333.088822] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3333.089600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3333.094045] Lustre: Skipped 1 previous similar message [ 3333.101692] Lustre: Skipped 1 previous similar message [ 3337.326032] LustreError: 124201:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3337.334233] LustreError: 124201:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3337.472785] Lustre: server umount lustre-MDT0000 complete [ 3338.208863] LustreError: 121728:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3338.225456] LustreError: 121728:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3340.749382] LustreError: 121712:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343537 with bad export cookie 10821578445762180361 [ 3340.770334] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3340.898898] LustreError: 124402:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3340.908186] LustreError: 124402:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3341.110360] Lustre: server umount lustre-MDT0001 complete [ 3350.923525] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 3354.053828] Key type lgssc unregistered [ 3354.437233] LNet: 124935:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3354.453648] LNetError: 124935:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3354.478806] LNet: Removed LNI 192.168.202.127@tcp [ 3355.303494] Key type .llcrypt unregistered [ 3355.305405] Key type ._llcrypt unregistered [ 3377.437528] Key type ._llcrypt registered [ 3377.440411] Key type .llcrypt registered [ 3377.614070] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 3378.581950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3378.613474] alg: No test for adler32 (adler32-zlib) [ 3379.645768] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 3379.962910] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 3381.680940] Key type lgssc registered [ 3383.062742] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3390.928842] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3390.961786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3392.341299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3396.116589] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3402.725021] LustreError: 126326:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3402.738674] LustreError: 126326:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3404.045345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3404.289546] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3407.418853] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3412.159661] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3415.524056] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3422.224170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3422.564823] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3428.179759] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3433.780617] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3436.918065] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3445.306234] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3448.874670] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3448.884168] Lustre: Skipped 2 previous similar messages [ 3452.589344] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3452.592649] Lustre: Skipped 2 previous similar messages [ 3453.535972] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3453.559861] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3461.964469] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.27@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3461.982986] Lustre: Skipped 2 previous similar messages [ 3471.926684] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.27@tcp 192.168.0.[2,10]@tcp [ 3475.319260] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.27/24@tcp in log lustre-MDT0000 [ 3475.327927] Lustre: Skipped 2 previous similar messages [ 3478.036456] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.27/24@tcp [ 3478.042269] Lustre: Skipped 1 previous similar message [ 3486.176215] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3486.198685] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3486.222703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3487.716637] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3487.722403] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3487.742551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3492.041549] LustreError: 129974:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3492.136837] Lustre: server umount lustre-OST0000 complete [ 3496.417649] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3496.428438] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3496.447848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3496.451449] Lustre: Skipped 2 previous similar messages [ 3498.465499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3498.472604] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3501.760416] LustreError: 130176:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3501.763841] LustreError: 130176:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3501.932956] Lustre: server umount lustre-MDT0000 complete [ 3503.589241] LustreError: 126322:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3503.597669] LustreError: 126322:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3505.090868] LustreError: 128624:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343702 with bad export cookie 16535803524461040527 [ 3505.095768] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3505.103778] LustreError: 128624:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3505.263070] LustreError: 130377:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3505.266660] LustreError: 130377:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3505.458645] Lustre: server umount lustre-MDT0001 complete [ 3515.367443] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 3517.823855] Key type lgssc unregistered [ 3518.043879] LNet: 130908:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3518.048394] LNetError: 130908:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3518.064262] LNet: Removed LNI 192.168.202.127@tcp [ 3518.826066] Key type .llcrypt unregistered [ 3518.827679] Key type ._llcrypt unregistered [ 3542.384339] Key type ._llcrypt registered [ 3542.389686] Key type .llcrypt registered [ 3542.531489] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 3543.421566] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3543.434040] alg: No test for adler32 (adler32-zlib) [ 3544.522564] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 3544.826226] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 3546.511264] Key type lgssc registered [ 3547.835549] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3555.966906] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3556.009960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3557.583989] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3557.626791] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.27/24@tcp [ 3557.626988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3563.088246] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3567.075898] LustreError: 132449:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3567.106492] LustreError: 132449:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3572.192588] LustreError: 132449:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3572.972580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3573.519297] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3573.582600] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.27/24@tcp [ 3573.666170] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3579.086346] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3585.765460] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3589.300764] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3597.052980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3597.760573] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3601.896098] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 3605.488674] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3612.195935] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3622.372196] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3622.377145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3622.379643] Lustre: Skipped 1 previous similar message [ 3622.380127] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3627.492155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3627.505130] Lustre: Skipped 1 previous similar message [ 3632.608991] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3632.621080] Lustre: Skipped 2 previous similar messages [ 3635.167165] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3635.210234] LustreError: 134756:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3635.365744] Lustre: server umount lustre-OST0000 complete [ 3640.291411] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3640.300148] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3640.324937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3642.863978] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3645.421163] LustreError: 134959:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3645.426370] LustreError: 134959:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3645.577660] Lustre: server umount lustre-MDT0000 complete [ 3647.978803] LustreError: 132450:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3647.978803] LustreError: 133392:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3650.095782] LustreError: 134670:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763343847 with bad export cookie 8749489036511214257 [ 3650.106148] LustreError: 134670:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3650.109472] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3650.327390] LustreError: 135160:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 3650.340884] LustreError: 135160:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3650.569852] Lustre: server umount lustre-MDT0001 complete [ 3661.969839] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 3664.813127] Key type lgssc unregistered [ 3665.193206] LNet: 135691:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3665.202659] LNetError: 135691:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3666.278276] LNet: Removed LNI 192.168.202.127@tcp [ 3667.125535] Key type .llcrypt unregistered [ 3667.135405] Key type ._llcrypt unregistered [ 3695.321711] Key type ._llcrypt registered [ 3695.323476] Key type .llcrypt registered [ 3695.434937] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 3710.474316] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 3711.825743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3711.833900] alg: No test for adler32 (adler32-zlib) [ 3712.880362] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 3713.145627] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 3714.863457] Key type lgssc registered [ 3715.956468] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3725.140872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3731.710102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3737.885557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3745.105090] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3753.207658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3753.291146] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3753.302868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3754.569471] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3754.597677] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3754.664583] Lustre: lustre-MDT0000: new disk, initializing [ 3754.814572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3754.837134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3759.746596] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3771.163886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3771.274087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3771.408081] Lustre: 139069:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3771.504324] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3771.514811] Lustre: Skipped 1 previous similar message [ 3771.662333] Lustre: lustre-MDT0001: new disk, initializing [ 3771.844228] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3771.925724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3771.946459] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3776.547958] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3783.990709] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3788.007477] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3795.543511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3795.643035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3795.860614] Lustre: lustre-OST0000: new disk, initializing [ 3795.866448] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3795.983654] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3801.705490] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3804.747282] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3804.767573] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3804.930894] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3809.523129] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3813.849966] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3814.128083] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3818.786395] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3819.069825] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3825.637246] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3825.637950] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3825.647648] LustreError: Skipped 1 previous similar message [ 3825.671243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3830.758073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3830.768741] Lustre: Skipped 2 previous similar messages [ 3835.362996] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3835.413720] LustreError: 142011:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3835.564615] Lustre: server umount lustre-OST0000 complete [ 3841.000090] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3841.014941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3841.020642] Lustre: Skipped 2 previous similar messages [ 3846.116500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3846.123403] Lustre: Skipped 2 previous similar messages [ 3851.235541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3851.242295] Lustre: Skipped 1 previous similar message [ 3853.279160] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3853.381767] LustreError: 142213:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3853.395216] LustreError: 142213:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3853.465281] Lustre: server umount lustre-MDT0000 complete [ 3856.758790] LustreError: 139061:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344053 with bad export cookie 9962002927500665408 [ 3856.766477] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3856.782908] LustreError: 139061:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3856.888859] LustreError: 142417:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3856.893176] LustreError: 142417:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3857.042703] Lustre: server umount lustre-MDT0001 complete [ 3865.301840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3865.921044] LustreError: 142893:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3865.948916] LustreError: 142893:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3866.071060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3870.012562] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3871.201448] LustreError: 142893:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3871.210721] LustreError: 142893:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3876.336200] LustreError: 142893:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3878.775280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3879.234758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3882.933336] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3888.710226] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3892.189851] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3898.991493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3899.470744] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3905.438687] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3911.138223] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3915.153108] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3915.321229] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3919.384464] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3919.608548] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3927.938496] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3928.000615] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3928.181180] Lustre: lustre-OST0001: new disk, initializing [ 3928.187676] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3928.287301] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3933.535383] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 3938.494030] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3938.508235] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3938.634201] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3941.779172] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3941.996611] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3945.631575] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3945.862839] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3949.816701] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff89a0845e8000.ost_server_uuid 50 [ 3951.591558] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff89a0845e8000.ost_server_uuid in FULL state after 0 sec [ 3959.271777] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3959.273701] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3959.276183] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3959.294141] LustreError: Skipped 1 previous similar message [ 3959.299764] Lustre: Skipped 1 previous similar message [ 3965.182251] LustreError: 147145:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3965.213921] LustreError: 147145:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3965.351912] Lustre: server umount lustre-OST0001 complete [ 3969.511356] LustreError: 144496:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3969.539883] LustreError: 144496:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3971.043106] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3971.046914] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3973.093399] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3974.634402] LustreError: 144497:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3974.665163] LustreError: 144497:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3975.754310] LustreError: 147347:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3975.764823] LustreError: 147347:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3975.888795] Lustre: server umount lustre-OST0000 complete [ 3979.745492] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3979.746570] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3979.769886] Lustre: Skipped 2 previous similar messages [ 3979.777168] Lustre: Skipped 7 previous similar messages [ 3985.739850] LustreError: 147548:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3985.747658] LustreError: 147548:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3985.896788] Lustre: server umount lustre-MDT0000 complete [ 3989.564569] LustreError: 142873:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344186 with bad export cookie 9962002927500666290 [ 3989.567175] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3989.579891] LustreError: 142873:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3990.006630] Lustre: server umount lustre-MDT0001 complete [ 4009.642351] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 4012.929725] Key type lgssc unregistered [ 4013.277984] LNet: 148739:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4013.293196] LNetError: 148739:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4013.308645] LNet: Removed LNI 192.168.202.127@tcp [ 4014.143755] Key type .llcrypt unregistered [ 4014.145409] Key type ._llcrypt unregistered [ 4049.507993] Key type ._llcrypt registered [ 4049.510023] Key type .llcrypt registered [ 4049.685895] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 4063.583758] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 4065.010342] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4065.029032] alg: No test for adler32 (adler32-zlib) [ 4066.166668] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 4066.398754] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 4068.106736] Key type lgssc registered [ 4069.289801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4077.656759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4084.164122] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4090.293701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4096.811257] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4104.302321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4104.366906] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4104.378823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4105.733561] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4105.785622] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4105.975901] Lustre: lustre-MDT0000: new disk, initializing [ 4106.159202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4106.187070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4111.661968] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 4124.497938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4124.560616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4124.622875] Lustre: 152694:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4124.683899] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4124.689532] Lustre: Skipped 1 previous similar message [ 4124.742286] Lustre: lustre-MDT0001: new disk, initializing [ 4124.839089] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4124.911140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4124.937201] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4129.257947] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 4136.261569] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4139.530781] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4146.644884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4146.740695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4146.964592] Lustre: lustre-OST0000: new disk, initializing [ 4146.968715] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4147.081923] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4152.818320] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 4156.993841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4157.008137] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4157.194478] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4160.098267] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4164.159707] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4164.422886] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4168.462495] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4168.705796] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4172.768343] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4172.770526] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4172.790643] LustreError: Skipped 1 previous similar message [ 4172.793294] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4172.795601] Lustre: Skipped 1 previous similar message [ 4172.817838] Lustre: Skipped 1 previous similar message [ 4176.768263] LustreError: 155634:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4176.952145] Lustre: server umount lustre-OST0000 complete [ 4181.479321] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4181.483290] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4181.489778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4183.013868] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4183.014278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4183.022317] Lustre: Skipped 1 previous similar message [ 4186.856469] LustreError: 155835:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4186.859707] LustreError: 155835:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4186.928479] Lustre: server umount lustre-MDT0000 complete [ 4189.954574] LustreError: 154439:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344387 with bad export cookie 14256161069431342771 [ 4189.961480] LustreError: 154439:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4189.965048] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4190.144031] LustreError: 156036:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4190.151728] LustreError: 156036:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4190.315093] Lustre: server umount lustre-MDT0001 complete [ 4197.268304] Lustre: DEBUG MARKER: == conf-sanity test 47a: server restart does not lose lru_resize ========================================================== 20:53:12 (1763344392) [ 4211.787799] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 4219.374272] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 4228.647643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4234.674491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4239.434242] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4245.030423] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4251.033460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4251.131177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4251.446894] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4251.474155] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4251.564168] Lustre: lustre-MDT0000: new disk, initializing [ 4251.677331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4251.696328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4255.623815] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 4265.236787] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4265.300820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4265.375411] Lustre: 159028:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4265.439469] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4265.442506] Lustre: Skipped 1 previous similar message [ 4265.486628] Lustre: lustre-MDT0001: new disk, initializing [ 4265.573228] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4265.608537] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4265.624620] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4269.756137] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 4275.735913] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4278.843831] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4284.346318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4284.382433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4284.570976] Lustre: lustre-OST0000: new disk, initializing [ 4284.577551] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4284.662734] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4289.126502] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 4290.120566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4290.133205] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4290.259127] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4296.765942] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4301.892756] Lustre: Failing over lustre-OST0000 [ 4303.849476] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4303.864158] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4303.876378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4304.012302] LustreError: 161469:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4304.025019] LustreError: 161469:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4304.084245] LustreError: 161178:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4304.110098] LustreError: 161178:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4304.267739] Lustre: server umount lustre-OST0000 complete [ 4321.675062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4321.902821] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4321.919644] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4322.911225] Lustre: 150905:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763344503/real 1763344503] req@00000000b56fd019 x1849000473990144/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1763344519 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4322.927097] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4322.934275] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4325.720026] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 4327.967547] Lustre: 150906:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763344508/real 1763344508] req@0000000041d5bd66 x1849000473991168/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1763344524 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4327.986339] Lustre: 150906:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4328.270134] Lustre: Failing over lustre-MDT0000 [ 4328.409937] LustreError: 162444:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4328.418479] LustreError: 162444:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4328.511802] Lustre: server umount lustre-MDT0000 complete [ 4329.654549] LustreError: 159813:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4332.002351] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4332.007101] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4332.017652] LustreError: 159039:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4332.383238] Lustre: 150906:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763344513/real 1763344513] req@000000009605a064 x1849000473992192/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1763344529 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4334.761886] LustreError: 159036:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4334.779296] LustreError: 159036:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4337.503220] Lustre: 150906:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763344518/real 1763344518] req@00000000a9331a7d x1849000473993216/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1763344534 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4339.886327] LustreError: 159034:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4339.894177] LustreError: 159034:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4345.726450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4345.822681] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4345.829514] Lustre: 159032:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.202.127@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 4345.966829] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 4346.034400] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4349.033872] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 4350.124678] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4351.466199] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4351.484075] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4351.549313] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4359.170492] LustreError: 163330:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4359.173677] LustreError: 163330:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4359.384932] Lustre: server umount lustre-OST0000 complete [ 4366.821198] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4366.821443] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4366.822167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4366.830815] Lustre: Skipped 3 previous similar messages [ 4372.037394] Lustre: server umount lustre-MDT0000 complete [ 4375.225077] LustreError: 159021:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344572 with bad export cookie 14256161069431344850 [ 4375.225723] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4375.234195] LustreError: 159021:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4385.791907] LustreError: 163734:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4385.802646] LustreError: 163734:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4385.991183] Lustre: server umount lustre-MDT0001 complete [ 4395.801318] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 4398.202726] Key type lgssc unregistered [ 4398.462447] LNet: 164271:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4398.468840] LNetError: 164271:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4398.486676] LNet: Removed LNI 192.168.202.127@tcp [ 4399.232194] Key type .llcrypt unregistered [ 4399.239318] Key type ._llcrypt unregistered [ 4430.502446] Key type ._llcrypt registered [ 4430.504832] Key type .llcrypt registered [ 4430.585126] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 4431.171824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4431.244075] alg: No test for adler32 (adler32-zlib) [ 4432.251407] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 4432.424669] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 4434.071232] Key type lgssc registered [ 4434.810226] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4439.732922] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4439.755431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4441.057831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4443.411447] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 4448.162789] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4451.142832] LustreError: 165800:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4451.150286] LustreError: 165800:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4451.155719] LustreError: 165800:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -5 [ 4451.256145] LustreError: 166322:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4451.361883] Lustre: server umount lustre-MDT0000 complete [ 4461.404436] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 4468.387567] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 4475.988458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4481.772262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4487.553823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4492.993658] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4498.069359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4498.138046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4498.340930] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4498.375531] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4498.431776] Lustre: lustre-MDT0000: new disk, initializing [ 4498.518582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4498.537544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4501.568567] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 4511.409259] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4511.486331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4511.572167] Lustre: 168964:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4511.638222] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4511.642236] Lustre: Skipped 1 previous similar message [ 4511.686844] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4511.727359] Lustre: lustre-MDT0001: new disk, initializing [ 4511.803743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4511.828609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4511.837554] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4515.380828] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 4522.610317] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4526.540315] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4533.026227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4533.098823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4533.370782] Lustre: lustre-OST0000: new disk, initializing [ 4533.377878] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4533.465880] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4537.741807] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 4541.513529] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4541.522401] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4541.691095] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4544.457462] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4548.434110] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4550.111846] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 20:59:05 (1763344745) [ 4780.511809] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4780.512038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4780.522398] Lustre: Skipped 2 previous similar messages [ 4780.530351] Lustre: Skipped 1 previous similar message [ 4785.633120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4785.635685] Lustre: Skipped 1 previous similar message [ 4785.864392] LustreError: 172500:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4785.878479] LustreError: 172500:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4785.977063] Lustre: server umount lustre-MDT0000 complete [ 4789.176293] LustreError: 169745:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763344986 with bad export cookie 3754598910609261890 [ 4789.183355] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4789.191659] LustreError: 169745:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4790.751747] LustreError: 172368:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4790.755931] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4790.760800] LustreError: 172368:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4790.786589] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4794.783664] LustreError: 168972:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4795.396938] LustreError: 172704:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4795.417174] LustreError: 172704:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4795.620501] Lustre: server umount lustre-MDT0001 complete [ 4814.303325] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4814.354577] LustreError: 172906:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4814.374627] LustreError: 172906:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4814.514454] Lustre: server umount lustre-OST0000 complete [ 4823.034719] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 4831.908945] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 4840.429483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4845.250121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4851.036755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4856.794460] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4862.626937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4862.698297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4862.904546] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4862.951528] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4863.025562] Lustre: lustre-MDT0000: new disk, initializing [ 4863.129704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4863.145964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4866.756115] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 4876.321919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4876.383886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4876.465120] Lustre: 175248:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4876.539985] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4876.548664] Lustre: Skipped 1 previous similar message [ 4876.611242] Lustre: lustre-MDT0001: new disk, initializing [ 4876.706912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4876.755188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4876.762358] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4880.411133] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 4887.143712] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4890.019049] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4895.684479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4895.761755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4895.970877] Lustre: lustre-OST0000: new disk, initializing [ 4895.974660] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4896.060569] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4900.662936] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 4903.532724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4903.537654] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4903.642327] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4907.495299] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4911.155161] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4911.370736] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4915.016181] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4915.205276] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4919.266326] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4919.270092] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4919.273456] LustreError: Skipped 1 previous similar message [ 4919.277855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4919.296679] Lustre: Skipped 1 previous similar message [ 4919.307882] Lustre: Skipped 1 previous similar message [ 4922.941987] LustreError: 178186:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4922.944148] LustreError: 178186:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4923.205362] Lustre: server umount lustre-OST0000 complete [ 4927.968224] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4927.979145] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4927.987778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4927.990663] Lustre: Skipped 1 previous similar message [ 4932.859284] LustreError: 178387:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4932.884469] LustreError: 178387:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4933.013316] Lustre: server umount lustre-MDT0000 complete [ 4934.631325] LustreError: 175259:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4934.641924] LustreError: 175259:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4936.261758] LustreError: 175240:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345133 with bad export cookie 3754598910609420720 [ 4936.264814] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4936.280824] LustreError: 175240:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4936.606259] Lustre: server umount lustre-MDT0001 complete [ 4944.925739] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 21:05:40 (1763345140) [ 4959.295451] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 4967.492704] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 4975.800247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4982.197698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4987.519477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4992.867493] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4999.354435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4999.406582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4999.659023] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4999.717271] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4999.782199] Lustre: lustre-MDT0000: new disk, initializing [ 4999.860451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4999.885258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5003.566725] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 5013.745515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5013.793933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5013.898267] Lustre: 181581:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5014.258519] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5017.388443] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 5024.270408] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5028.015364] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5034.813373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5034.899888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5035.134317] Lustre: lustre-OST0000: new disk, initializing [ 5035.136430] Lustre: Skipped 1 previous similar message [ 5035.140866] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5035.146818] Lustre: Skipped 2 previous similar messages [ 5040.598677] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 5044.326927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5044.336370] Lustre: Skipped 1 previous similar message [ 5044.338854] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5044.498133] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5048.011827] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5054.944072] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5054.947382] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5054.951716] LustreError: Skipped 1 previous similar message [ 5054.953487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5054.961111] Lustre: Skipped 2 previous similar messages [ 5054.968678] Lustre: Skipped 3 previous similar messages [ 5060.167079] LustreError: 184126:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5060.170082] LustreError: 184126:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5060.399395] Lustre: server umount lustre-OST0000 complete [ 5065.189715] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5065.211204] Lustre: Skipped 1 previous similar message [ 5069.898721] Lustre: server umount lustre-MDT0000 complete [ 5070.305387] LustreError: 181591:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5070.319431] LustreError: 181591:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5073.419813] LustreError: 181573:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345270 with bad export cookie 3754598910609421602 [ 5073.422331] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5073.435991] LustreError: 181573:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5073.820070] Lustre: server umount lustre-MDT0001 complete [ 5080.465390] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 21:07:56 (1763345276) [ 5093.451709] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 5102.823346] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 5112.270694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5119.400935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5126.359025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5134.204372] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5142.167425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5142.225567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5142.388235] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5142.419609] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5142.474547] Lustre: lustre-MDT0000: new disk, initializing [ 5142.559250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5142.562014] Lustre: Skipped 2 previous similar messages [ 5142.583820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5146.671873] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 5157.052323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5157.118765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5157.227841] Lustre: 187523:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5157.334943] LustreError: 187534:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5157.631483] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5161.171092] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 5167.633450] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5170.858563] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5177.328917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5177.400429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5183.774144] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5183.875660] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 5191.783437] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5199.331496] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5199.333236] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5199.352815] LustreError: Skipped 1 previous similar message [ 5199.355986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5199.363648] Lustre: Skipped 4 previous similar messages [ 5204.099189] LustreError: 190069:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5204.115225] LustreError: 190069:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5204.298550] Lustre: server umount lustre-OST0000 complete [ 5209.055975] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5214.397054] Lustre: server umount lustre-MDT0000 complete [ 5214.688498] LustreError: 188659:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5214.710669] LustreError: 188659:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5218.186671] LustreError: 187516:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345415 with bad export cookie 3754598910609422687 [ 5218.196984] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5218.262047] LustreError: 187516:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5218.624610] Lustre: server umount lustre-MDT0001 complete [ 5228.367921] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 5231.427826] Key type lgssc unregistered [ 5231.759874] LNet: 191004:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5231.776393] LNetError: 191004:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5232.810103] LNet: Removed LNI 192.168.202.127@tcp [ 5233.899472] Key type .llcrypt unregistered [ 5233.901374] Key type ._llcrypt unregistered [ 5259.862696] Key type ._llcrypt registered [ 5259.866221] Key type .llcrypt registered [ 5259.979044] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 5261.264099] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5261.379216] alg: No test for adler32 (adler32-zlib) [ 5262.480821] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 5262.780441] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 5264.511449] Key type lgssc registered [ 5265.879348] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5275.751612] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5275.807529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5277.875339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5283.779902] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 5288.416914] LustreError: 192403:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5288.422305] LustreError: 192403:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5293.540565] LustreError: 192402:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5295.212487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5295.918918] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5300.530856] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 5306.031361] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5309.483811] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5315.909520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5316.205481] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5322.873834] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 5323.385585] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5330.320430] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5352.419555] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5352.440889] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5352.456811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5354.465079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5354.470031] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5354.493244] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5355.880142] LustreError: 194670:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5356.063403] Lustre: server umount lustre-OST0000 complete [ 5362.661516] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5362.668528] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5362.699362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5362.705394] Lustre: Skipped 1 previous similar message [ 5364.707203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5364.708186] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5366.333213] LustreError: 194872:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5366.342382] LustreError: 194872:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5366.477905] Lustre: server umount lustre-MDT0000 complete [ 5369.824519] LustreError: 192403:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5369.840090] LustreError: 192403:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5370.333524] LustreError: 192383:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345567 with bad export cookie 15582743041993312570 [ 5370.353765] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5370.508488] LustreError: 195073:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5370.514576] LustreError: 195073:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5370.765161] Lustre: server umount lustre-MDT0001 complete [ 5381.885406] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 5384.783699] Key type lgssc unregistered [ 5385.080333] LNet: 195605:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5385.102333] LNetError: 195605:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5385.125550] LNet: Removed LNI 192.168.202.127@tcp [ 5386.019095] Key type .llcrypt unregistered [ 5386.020949] Key type ._llcrypt unregistered [ 5410.744068] Key type ._llcrypt registered [ 5410.745762] Key type .llcrypt registered [ 5410.849540] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 5412.184134] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5412.219588] alg: No test for adler32 (adler32-zlib) [ 5413.393527] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 5413.763246] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 5415.431194] Key type lgssc registered [ 5416.461805] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5423.641562] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5423.653501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5425.051165] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5428.994470] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 5435.362744] LustreError: 197004:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5435.381609] LustreError: 197004:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5437.038323] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5437.481703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5441.106274] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 5446.417635] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5449.767728] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5456.062701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5456.486034] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5461.770268] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 5466.805085] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5469.816966] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5477.586458] Lustre: lustre-OST0000: Not available for connect from 192.168.202.27@tcp (stopping) [ 5478.377260] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5478.381987] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5478.390177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5480.426082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5480.426765] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5482.670979] Lustre: lustre-OST0000: Not available for connect from 192.168.202.27@tcp (stopping) [ 5482.673582] Lustre: Skipped 1 previous similar message [ 5482.738110] LustreError: 199253:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5482.886284] Lustre: server umount lustre-OST0000 complete [ 5488.054640] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff89a085c69000.ost_server_uuid 50 [ 5490.058354] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5503.970696] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5503.975352] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5503.989211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5509.166755] LustreError: 199624:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5509.170475] LustreError: 199624:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5509.304489] Lustre: server umount lustre-MDT0000 complete [ 5510.123795] LustreError: 197001:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5510.155270] LustreError: 197001:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5512.597537] LustreError: 199524:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345709 with bad export cookie 16703288990071624073 [ 5512.601893] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5512.609682] LustreError: 199524:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5512.750311] LustreError: 199825:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5512.757316] LustreError: 199825:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5512.911045] Lustre: server umount lustre-MDT0001 complete [ 5520.445392] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 21:15:16 (1763345716) [ 5526.507738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5526.922428] LustreError: 200497:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5526.992162] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5531.092203] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 5532.130621] LustreError: 200497:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5537.251190] LustreError: 200497:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5539.051587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5539.484988] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5542.939506] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 5548.438334] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5552.394485] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5559.465964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5559.817695] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5564.828544] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 5569.422843] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5572.086305] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5576.070408] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5576.183699] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5576.335336] Lustre: lustre-OST0001: new disk, initializing [ 5576.349988] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5576.475247] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5582.192970] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 5586.531777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5586.541313] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5586.622929] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5589.909821] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5594.592249] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5594.599064] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5594.620198] Lustre: Skipped 1 previous similar message [ 5594.634067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5594.641134] Lustre: Skipped 2 previous similar messages [ 5598.656088] LustreError: 203702:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5598.663897] LustreError: 203702:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5598.722745] Lustre: server umount lustre-OST0000 complete [ 5600.947159] LustreError: 203517:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5604.856606] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5605.074746] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5610.432625] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5610.871779] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5617.636887] LustreError: 202096:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5617.663308] LustreError: 202096:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 5627.883020] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5627.892738] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5627.900249] Lustre: Skipped 2 previous similar messages [ 5627.909060] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5627.914403] Lustre: Skipped 4 previous similar messages [ 5630.087885] LustreError: 204425:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5630.091025] LustreError: 204425:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5630.339384] Lustre: server umount lustre-OST0001 complete [ 5637.088588] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5637.101569] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5640.868343] LustreError: 204626:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5640.878604] LustreError: 204626:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5640.995917] Lustre: server umount lustre-MDT0000 complete [ 5645.427244] LustreError: 200478:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345842 with bad export cookie 16703288990071625144 [ 5645.455563] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5646.066989] Lustre: server umount lustre-MDT0001 complete [ 5652.275905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5656.404706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5660.023738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5663.870175] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5670.570766] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 21:17:46 (1763345866) [ 5677.204337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5677.499458] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5677.537657] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5677.751076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5681.392279] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 5690.456868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5690.570257] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5690.591741] Lustre: 206319:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5694.709518] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 5699.544707] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5703.389702] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5710.316611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5710.533845] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5716.480597] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 5718.702684] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 5722.656492] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5729.959138] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5730.113887] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5736.572264] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 5739.180363] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 5741.939856] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5747.931817] Lustre: lustre-OST0000: Not available for connect from 192.168.202.27@tcp (stopping) [ 5747.937127] Lustre: Skipped 3 previous similar messages [ 5749.218649] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5749.236659] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5749.257733] Lustre: Skipped 1 previous similar message [ 5750.722653] LustreError: 209521:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5750.743844] LustreError: 209521:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5750.867682] Lustre: server umount lustre-OST0000 complete [ 5753.031371] LustreError: 207951:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5753.066968] LustreError: 207951:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 5770.212139] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5770.219901] LustreError: Skipped 1 previous similar message [ 5772.602289] Lustre: server umount lustre-OST0001 complete [ 5782.922813] LustreError: 209947:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5782.956670] LustreError: 209947:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5783.179762] Lustre: server umount lustre-MDT0000 complete [ 5786.986410] LustreError: 206314:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763345984 with bad export cookie 16703288990071626551 [ 5786.995071] LustreError: 206314:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5786.999982] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5787.425755] Lustre: server umount lustre-MDT0001 complete [ 5793.673816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5797.396708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5800.857257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5804.368364] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5811.080438] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 21:20:06 (1763346006) [ 5825.254379] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 5833.691246] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 5843.153210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5848.625395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5854.211228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5859.524586] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5865.688361] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5865.762256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5866.029793] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5866.180286] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5866.233522] Lustre: lustre-MDT0000: new disk, initializing [ 5866.335713] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5866.342538] Lustre: Skipped 3 previous similar messages [ 5866.359290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5871.367349] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 5882.710564] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5882.797639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5882.893267] Lustre: 213973:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5882.946790] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5882.951833] Lustre: Skipped 1 previous similar message [ 5883.118577] Lustre: lustre-MDT0001: new disk, initializing [ 5883.393523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5883.413170] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5888.165338] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 5895.981973] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5899.852598] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5905.889592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5905.964293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5906.172722] Lustre: lustre-OST0000: new disk, initializing [ 5906.176721] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5911.643593] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 5915.708405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5915.713380] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5915.799211] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5918.905488] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5922.947810] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5923.183821] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5926.922416] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5927.132821] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5931.494313] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5931.494343] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5931.499896] LustreError: Skipped 2 previous similar messages [ 5931.514987] Lustre: Skipped 5 previous similar messages [ 5931.515733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5931.524377] Lustre: Skipped 8 previous similar messages [ 5943.263822] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5943.336185] LustreError: 216914:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5943.338839] LustreError: 216914:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5943.530627] Lustre: server umount lustre-OST0000 complete [ 5953.629756] Lustre: server umount lustre-MDT0000 complete [ 5957.090620] LustreError: 213981:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5957.109513] LustreError: 213981:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 5957.270275] LustreError: 213966:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346154 with bad export cookie 16703288990071628336 [ 5957.282385] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5957.599852] Lustre: server umount lustre-MDT0001 complete [ 5966.209536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5970.819508] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 5979.002063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5983.815901] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 5989.057812] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5993.095579] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6000.832995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6001.186475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6001.193463] Lustre: Skipped 4 previous similar messages [ 6007.056796] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6012.437295] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6016.114219] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6016.375704] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6020.211491] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6020.407851] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6024.237224] Lustre: server umount lustre-OST0000 complete [ 6030.643537] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6040.031223] Lustre: 196314:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763346221/real 1763346221] req@0000000021830953 x1849001886572032/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1763346237 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6040.063275] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6040.078675] Lustre: Skipped 3 previous similar messages [ 6042.079900] Lustre: 196314:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763346223/real 1763346223] req@00000000a1f8559e x1849001886573056/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1763346239 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6042.112054] Lustre: 196314:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 6046.326706] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6048.159194] Lustre: 196314:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763346229/real 1763346229] req@000000006cd15da0 x1849001886573952/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1763346245 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6048.181094] Lustre: 196314:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6053.279693] Lustre: 196314:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763346234/real 1763346234] req@0000000058e5902a x1849001886575488/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1763346250 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6075.370796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6081.131709] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6084.968245] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6084.992065] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6086.016770] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6087.084219] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6087.104379] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6093.870777] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 6 sec [ 6097.387453] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6097.391635] Lustre: Skipped 8 previous similar messages [ 6098.548129] LustreError: 222258:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6098.555574] LustreError: 222258:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 6098.721572] Lustre: server umount lustre-OST0000 complete [ 6102.498286] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6102.512586] LustreError: Skipped 1 previous similar message [ 6116.321253] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6116.610610] Lustre: server umount lustre-MDT0000 complete [ 6116.835827] LustreError: 217795:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6116.863278] LustreError: 217795:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6120.070222] LustreError: 217776:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346317 with bad export cookie 16703288990071629218 [ 6120.073698] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6120.083276] LustreError: 217776:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6120.475393] Lustre: server umount lustre-MDT0001 complete [ 6129.711694] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 21:25:25 (1763346325) [ 6137.451494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6143.538681] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6154.186253] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6159.026674] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6164.442249] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6168.409461] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6174.466169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6180.792538] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6186.811489] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6191.009931] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6191.298644] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6195.169114] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6195.416933] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6201.201624] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6201.281797] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6201.391656] Lustre: lustre-OST0001: new disk, initializing [ 6201.399555] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6207.128695] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6211.695298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6211.708136] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6211.834181] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6213.842613] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6217.560937] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6217.834313] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6221.715316] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6221.905519] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6227.433265] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6227.445939] Lustre: Skipped 6 previous similar messages [ 6238.175173] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6238.300425] Lustre: server umount lustre-OST0001 complete [ 6243.886323] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6244.102540] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6248.476464] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6248.787085] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6277.360921] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6277.561524] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6277.568927] Lustre: Skipped 5 previous similar messages [ 6282.758286] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6282.780522] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6282.786238] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6282.889527] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6289.147461] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6290.658795] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6292.965930] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6292.969658] LustreError: Skipped 2 previous similar messages [ 6298.354447] Lustre: server umount lustre-OST0001 complete [ 6328.287670] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6328.482835] Lustre: server umount lustre-OST0000 complete [ 6338.501877] Lustre: server umount lustre-MDT0000 complete [ 6342.398527] LustreError: 223316:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346539 with bad export cookie 16703288990071630184 [ 6342.407332] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6342.407496] LustreError: 223316:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6348.892763] Lustre: server umount lustre-MDT0001 complete [ 6353.892899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6357.565784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6361.041603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6364.801884] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6371.118754] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 21:29:26 (1763346566) [ 6377.502592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6377.780519] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6377.808124] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6381.625883] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6389.669909] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6389.791602] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6389.832623] Lustre: 231325:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6394.298566] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6400.366043] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6404.011268] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6412.233885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6412.396957] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6417.495321] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6422.007632] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6430.770955] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6430.872772] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6436.327596] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6442.120881] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6443.788986] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6448.082213] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6448.334805] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6453.110975] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6453.438122] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6456.915674] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff89a093d63800.ost_server_uuid 50 [ 6458.200258] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff89a093d63800.ost_server_uuid in FULL state after 0 sec [ 6459.079260] Lustre: Permanently deactivating lustre-OST0001 [ 6459.095598] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6461.754955] Lustre: Permanently reactivating lustre-OST0001 [ 6466.538247] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6466.545219] Lustre: Skipped 26 previous similar messages [ 6471.285089] LustreError: 235316:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 6471.297269] LustreError: 235316:0:(obd_class.h:479:obd_check_dev()) Skipped 47 previous similar messages [ 6471.368554] Lustre: server umount lustre-OST0001 complete [ 6471.661471] LustreError: 232951:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6471.668112] LustreError: 232951:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 37 previous similar messages [ 6480.803824] Lustre: server umount lustre-OST0000 complete [ 6487.009902] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6487.035332] Lustre: Skipped 11 previous similar messages [ 6490.590694] Lustre: server umount lustre-MDT0000 complete [ 6494.562673] LustreError: 231319:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346691 with bad export cookie 16703288990071631472 [ 6494.566093] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6494.593501] LustreError: 231319:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6495.073920] Lustre: server umount lustre-MDT0001 complete [ 6502.896096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6506.902858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6510.442819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6513.710461] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6520.589889] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 21:31:56 (1763346716) [ 6521.674303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6528.622316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6528.992785] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6529.058399] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6529.068542] Lustre: Skipped 5 previous similar messages [ 6533.439954] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6542.024472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6542.103588] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6542.136363] Lustre: 237476:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6545.529516] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6550.212243] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6552.843881] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6559.114573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6559.201958] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6559.405567] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6559.423115] Lustre: Permanently deactivating lustre-OST0000 [ 6559.435774] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6565.161895] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6566.468835] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6570.950493] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6578.068840] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6578.184727] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6583.747949] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6583.987869] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6589.474377] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6591.348524] Lustre: Permanently reactivating lustre-OST0000 [ 6591.385618] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6591.396129] Lustre: Skipped 2 previous similar messages [ 6597.197372] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6597.200845] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6597.203871] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 31 seconds. I think it's dead, and I am evicting it. exp 00000000eefabb94, cur 1763346794 deadline 1763346793 last 1763346763 [ 6597.236096] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6597.236104] Lustre: Skipped 1 previous similar message [ 6597.274974] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6597.289448] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6597.407542] LustreError: lustre-OST0000-osc-MDT0000: operation ost_create to node 0@lo failed: rc = -107 [ 6597.411729] LustreError: Skipped 7 previous similar messages [ 6607.745418] Lustre: server umount lustre-OST0001 complete [ 6617.526078] Lustre: server umount lustre-OST0000 complete [ 6627.246222] Lustre: server umount lustre-MDT0000 complete [ 6630.817340] LustreError: 240628:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346828 with bad export cookie 16703288990071633124 [ 6630.820232] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6630.824865] LustreError: 240628:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6631.230609] Lustre: server umount lustre-MDT0001 complete [ 6639.480474] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 6642.001528] Key type lgssc unregistered [ 6642.194852] LNet: 241897:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6642.204175] LNetError: 241897:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6642.217210] LNet: Removed LNI 192.168.202.127@tcp [ 6642.979954] Key type .llcrypt unregistered [ 6642.986724] Key type ._llcrypt unregistered [ 6665.508041] Key type ._llcrypt registered [ 6665.516424] Key type .llcrypt registered [ 6665.614109] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 6666.399511] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6666.472113] alg: No test for adler32 (adler32-zlib) [ 6667.515862] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 6667.666321] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 6669.335275] Key type lgssc registered [ 6670.337341] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6673.344808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6680.679527] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6680.714280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6682.260346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6687.636647] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6691.813721] LustreError: 243345:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6691.851626] LustreError: 243345:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6695.927490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6696.008553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6696.075316] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6696.097875] Lustre: 243332:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6696.104180] Lustre: Permanently deactivating lustre-MDT0001 [ 6696.120335] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6696.361487] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6700.154953] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6704.156568] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6705.608915] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6709.564399] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6716.169423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6716.467300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6721.587644] LustreError: 244968:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6721.609993] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:193) [ 6722.215429] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6726.625593] LustreError: 244965:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6727.668384] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6727.865698] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6732.785411] LustreError: 244968:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6732.809250] LustreError: 244968:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6734.996951] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6735.287232] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6740.478175] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6740.485986] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6741.433843] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6747.266260] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6751.421113] Lustre: Permanently reactivating lustre-MDT0001 [ 6751.453681] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6751.457080] Lustre: Skipped 2 previous similar messages [ 6753.471985] Lustre: lustre-MDT0001: Client e5444618-5c67-4add-831b-a516f145d02d (at 192.168.202.27@tcp) reconnecting [ 6757.619854] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6757.996340] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6758.005154] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 6758.017147] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 6758.024402] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6759.387844] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6764.443934] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6764.678882] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6768.127844] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6768.337899] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6770.673701] Lustre: Permanently deactivating lustre-MDT0001 [ 6770.695084] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6770.697781] Lustre: Skipped 2 previous similar messages [ 6781.033408] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6786.640264] LustreError: 247465:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 6786.714901] Lustre: server umount lustre-MDT0000 complete [ 6788.577930] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6788.579224] LustreError: 243340:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6788.598189] Lustre: Skipped 3 previous similar messages [ 6788.606291] LustreError: 243340:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6790.189810] LustreError: 246456:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763346987 with bad export cookie 1092212397473440186 [ 6790.193257] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6790.195966] LustreError: 246456:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6790.327903] LustreError: 247666:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 6790.338516] LustreError: 247666:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6790.498342] Lustre: server umount lustre-MDT0001 complete [ 6794.139094] LustreError: 247867:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 6794.147419] LustreError: 247867:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6794.192078] Lustre: server umount lustre-OST0000 complete [ 6805.936033] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 21:36:41 (1763347001) [ 6817.597466] LustreError: 248873:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6817.609367] LustreError: 248873:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6817.834642] Lustre: server umount lustre-OST0001 complete [ 6825.098803] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 6835.180418] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 6845.419943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6851.951983] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6858.807527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6865.232770] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6871.965653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6872.036908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6872.266166] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6872.273486] Lustre: Skipped 2 previous similar messages [ 6872.310687] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6872.366148] Lustre: lustre-MDT0000: new disk, initializing [ 6872.470887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6872.501422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6876.443578] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6887.443080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6887.529981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6887.608128] Lustre: 251060:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6887.671109] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6887.673771] Lustre: Skipped 1 previous similar message [ 6887.783776] Lustre: lustre-MDT0001: new disk, initializing [ 6887.939843] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6887.996552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6888.015054] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6892.240575] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6899.868989] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6903.541182] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6910.211554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6910.300908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6910.542850] Lustre: lustre-OST0000: new disk, initializing [ 6910.551548] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6910.627107] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6916.163801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6916.173478] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6916.290205] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6917.115385] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6926.969422] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6929.103806] LustreError: 253388:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6937.270933] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6937.322519] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6937.450875] Lustre: lustre-OST0001: new disk, initializing [ 6937.460061] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6939.175130] LustreError: 253388:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6939.207898] LustreError: 253388:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6943.627335] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 6945.410480] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6945.418535] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6945.507096] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6949.314443] LustreError: 253388:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6952.006446] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6953.664178] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6956.005139] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6956.014479] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6956.028885] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6956.035850] Lustre: Skipped 1 previous similar message [ 6958.798475] Lustre: lustre-OST0001: Not available for connect from 192.168.202.27@tcp (stopping) [ 6958.801018] Lustre: Skipped 1 previous similar message [ 6961.131478] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6961.147963] Lustre: Skipped 1 previous similar message [ 6963.886278] Lustre: lustre-OST0001: Not available for connect from 192.168.202.27@tcp (stopping) [ 6969.008516] Lustre: lustre-OST0001: Not available for connect from 192.168.202.27@tcp (stopping) [ 6969.022118] Lustre: Skipped 2 previous similar messages [ 6970.335319] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6970.376089] LustreError: 254552:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 6970.379715] LustreError: 254552:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6970.445076] Lustre: server umount lustre-OST0001 complete [ 6971.367538] LustreError: 252788:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6971.381817] LustreError: 252788:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6986.724495] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6986.732157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6986.744023] Lustre: Skipped 1 previous similar message [ 6991.853328] LustreError: 252790:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6991.861424] LustreError: 252790:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 6992.081106] LustreError: 254778:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6992.095404] LustreError: 254778:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6992.230668] Lustre: server umount lustre-OST0000 complete [ 6996.966528] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7007.203657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7007.209100] Lustre: Skipped 7 previous similar messages [ 7010.271214] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7010.354522] LustreError: 254979:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 7010.357544] LustreError: 254979:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7010.435211] Lustre: server umount lustre-MDT0000 complete [ 7013.815972] LustreError: 251053:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347211 with bad export cookie 1092212397473443224 [ 7013.820725] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7013.825823] LustreError: 251053:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7028.191495] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7028.611240] Lustre: server umount lustre-MDT0001 complete [ 7039.757648] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 7042.902902] Key type lgssc unregistered [ 7043.129828] LNet: 255718:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7043.137319] LNetError: 255718:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7043.159287] LNet: Removed LNI 192.168.202.127@tcp [ 7043.963447] Key type .llcrypt unregistered [ 7043.970381] Key type ._llcrypt unregistered [ 7079.705276] Key type ._llcrypt registered [ 7079.711928] Key type .llcrypt registered [ 7079.841882] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 7097.574481] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 7098.681066] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7098.840306] alg: No test for adler32 (adler32-zlib) [ 7099.873875] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 7100.087721] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 7101.784125] Key type lgssc registered [ 7103.210129] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7113.230294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7120.172740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7126.715633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7133.846085] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7141.900167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7141.982880] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7142.000961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7143.334732] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7143.368665] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7143.421659] Lustre: lustre-MDT0000: new disk, initializing [ 7143.539613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7143.564914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7147.927535] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7159.674556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7159.765563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7159.834935] Lustre: 259674:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7159.905553] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7159.908822] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7159.915941] Lustre: Skipped 1 previous similar message [ 7159.973950] Lustre: lustre-MDT0001: new disk, initializing [ 7160.110480] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7160.187213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7160.204290] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7164.562310] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7171.879752] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7175.581799] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7182.201779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7182.315489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7182.638532] Lustre: lustre-OST0000: new disk, initializing [ 7182.644416] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7182.742741] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7188.668916] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7190.626096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7190.650635] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7190.702581] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7195.977693] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7199.614880] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7199.801360] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7204.126543] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7204.348929] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7206.370288] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7206.376306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7206.414807] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7206.429320] LustreError: Skipped 1 previous similar message [ 7211.494693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7211.514660] Lustre: Skipped 2 previous similar messages [ 7212.493179] LustreError: 262612:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7212.615619] Lustre: server umount lustre-OST0000 complete [ 7216.610753] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7216.617671] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7216.621038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7216.640728] Lustre: Skipped 2 previous similar messages [ 7221.733692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7221.745588] Lustre: Skipped 2 previous similar messages [ 7222.481574] LustreError: 262814:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7222.484927] LustreError: 262814:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7222.641589] Lustre: server umount lustre-MDT0000 complete [ 7226.991529] LustreError: 259665:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347424 with bad export cookie 1749911348961413702 [ 7226.996884] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7227.011501] LustreError: 259665:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7227.162621] LustreError: 263016:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7227.164857] LustreError: 263016:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7227.347839] Lustre: server umount lustre-MDT0001 complete [ 7236.439934] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 21:43:52 (1763347432) [ 7243.217738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7243.728864] LustreError: 263686:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7243.736547] LustreError: 263686:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7243.841085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7248.679606] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7248.865897] LustreError: 263686:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7253.999935] LustreError: 263686:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7257.385529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7257.771766] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7262.159241] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7267.974841] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7271.965971] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7280.229977] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7280.608030] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7286.046600] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7291.740825] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7344.096826] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7344.104393] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7344.121718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7349.899119] LustreError: 266011:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7349.901896] LustreError: 266011:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7350.075703] Lustre: server umount lustre-OST0000 complete [ 7354.379816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7365.577835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7365.946520] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7371.171634] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7373.098150] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7373.105815] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7373.105978] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7376.444268] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7419.362851] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7419.372369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7419.398944] Lustre: Skipped 2 previous similar messages [ 7419.403248] Lustre: Skipped 3 previous similar messages [ 7422.585715] LustreError: 267684:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7422.603619] LustreError: 267684:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7422.888108] Lustre: server umount lustre-OST0000 complete [ 7426.529212] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7426.545111] LustreError: Skipped 1 previous similar message [ 7426.553268] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7432.729644] LustreError: 267885:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7432.758710] LustreError: 267885:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7432.892719] Lustre: server umount lustre-MDT0000 complete [ 7434.721163] LustreError: 263685:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7434.740058] LustreError: 263685:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7436.601096] LustreError: 263666:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347633 with bad export cookie 1749911348961414584 [ 7436.607464] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7436.619635] LustreError: 263666:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7436.900254] Lustre: server umount lustre-MDT0001 complete [ 7449.141709] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 7451.830497] Key type lgssc unregistered [ 7452.154487] LNet: 268620:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7452.170115] LNetError: 268620:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7452.192678] LNet: Removed LNI 192.168.202.127@tcp [ 7453.253067] Key type .llcrypt unregistered [ 7453.254806] Key type ._llcrypt unregistered [ 7475.445416] Key type ._llcrypt registered [ 7475.454181] Key type .llcrypt registered [ 7475.559562] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 7476.460731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7476.566277] alg: No test for adler32 (adler32-zlib) [ 7477.576552] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 7477.869275] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 7479.568099] Key type lgssc registered [ 7480.915175] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7487.842397] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7487.858094] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7489.215945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7492.427718] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7499.746832] LustreError: 270009:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7499.756578] LustreError: 270009:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7499.991410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7500.290103] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7503.916390] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7509.580727] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7513.740825] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7519.938702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7520.224054] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7525.588385] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7530.824375] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7534.587182] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7555.041774] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7555.043878] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7555.056450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7555.059194] Lustre: Skipped 1 previous similar message [ 7560.163473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7560.169400] Lustre: Skipped 1 previous similar message [ 7561.082509] LustreError: 273029:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7561.230570] Lustre: server umount lustre-OST0000 complete [ 7565.282780] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7565.288462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7565.295645] Lustre: Skipped 2 previous similar messages [ 7570.399914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7570.409254] Lustre: Skipped 2 previous similar messages [ 7570.421252] LustreError: 273230:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7570.426236] LustreError: 273230:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7570.513222] Lustre: server umount lustre-MDT0000 complete [ 7573.631541] LustreError: 273233:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347770 with bad export cookie 8353295132317446594 [ 7573.638885] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7573.647149] LustreError: 273233:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7573.767900] LustreError: 273435:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7573.772728] LustreError: 273435:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7573.984952] Lustre: server umount lustre-MDT0001 complete [ 7582.407582] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 7584.476498] Key type lgssc unregistered [ 7584.792931] LNet: 273966:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7584.802654] LNetError: 273966:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7585.837016] LNet: Removed LNI 192.168.202.127@tcp [ 7586.518170] Key type .llcrypt unregistered [ 7586.522359] Key type ._llcrypt unregistered [ 7601.422506] Key type ._llcrypt registered [ 7601.423948] Key type .llcrypt registered [ 7601.529351] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 7602.095287] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7602.112079] alg: No test for adler32 (adler32-zlib) [ 7603.146685] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 7603.304831] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 7604.935190] Key type lgssc registered [ 7605.904522] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7612.516666] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7612.533127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7614.091733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7617.549437] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7624.672329] LustreError: 275168:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7624.687600] LustreError: 275168:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7625.700818] LustreError: 275168:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7626.959076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7627.266452] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7631.090269] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7636.868474] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7640.843301] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7647.246593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7647.692210] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7653.458433] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7658.758909] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7660.024409] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7675.362860] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7675.366482] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7675.372723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7675.403566] Lustre: Skipped 1 previous similar message [ 7678.543980] LustreError: 277561:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7678.654873] Lustre: server umount lustre-OST0000 complete [ 7683.552531] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7683.555428] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7683.588901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7683.591843] Lustre: Skipped 1 previous similar message [ 7685.599905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7685.604532] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7688.820904] LustreError: 277763:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7688.843386] LustreError: 277763:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7688.926476] Lustre: server umount lustre-MDT0000 complete [ 7690.723979] LustreError: 277279:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7692.811391] LustreError: 275150:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763347890 with bad export cookie 8222544863445915864 [ 7692.825451] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7692.837235] LustreError: 275150:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7693.011916] LustreError: 277964:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7693.021331] LustreError: 277964:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7693.312934] Lustre: server umount lustre-MDT0001 complete [ 7703.430721] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 7706.022023] Key type lgssc unregistered [ 7706.320081] LNet: 278496:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7706.336331] LNetError: 278496:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7706.355911] LNet: Removed LNI 192.168.202.127@tcp [ 7707.185347] Key type .llcrypt unregistered [ 7707.187110] Key type ._llcrypt unregistered [ 7731.329434] Key type ._llcrypt registered [ 7731.331126] Key type .llcrypt registered [ 7731.433993] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 7732.147918] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7732.158466] alg: No test for adler32 (adler32-zlib) [ 7733.293486] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 7733.526641] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 7735.247461] Key type lgssc registered [ 7736.326298] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7743.870485] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7743.899459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7745.268746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7749.112622] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7755.745298] LustreError: 279894:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7755.761885] LustreError: 279894:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7757.543616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7757.819165] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7760.761584] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7764.359250] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7767.037675] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7772.327587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7772.635277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7777.219233] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7781.696444] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7790.067225] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7812.576350] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7812.588084] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7812.615206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7815.653096] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7815.664057] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7815.683844] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7817.172421] LustreError: 282934:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7817.334773] Lustre: server umount lustre-OST0000 complete [ 7820.770817] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7820.778470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7820.782347] Lustre: Skipped 2 previous similar messages [ 7825.890829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7825.902466] Lustre: Skipped 1 previous similar message [ 7831.011487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7831.014452] Lustre: Skipped 1 previous similar message [ 7835.103229] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7835.188150] LustreError: 283135:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7835.202159] LustreError: 283135:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7835.301513] Lustre: server umount lustre-MDT0000 complete [ 7836.129052] LustreError: 279895:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7836.156750] LustreError: 279895:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7838.790048] LustreError: 279875:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763348036 with bad export cookie 14834162679061144778 [ 7838.795510] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7838.947270] LustreError: 283338:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7838.960722] LustreError: 283338:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7839.125447] Lustre: server umount lustre-MDT0001 complete [ 7848.881330] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 7850.953308] Key type lgssc unregistered [ 7851.166481] LNet: 283870:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7851.182711] LNetError: 283870:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7851.197766] LNet: Removed LNI 192.168.202.127@tcp [ 7851.951451] Key type .llcrypt unregistered [ 7851.964398] Key type ._llcrypt unregistered [ 7866.527395] Key type ._llcrypt registered [ 7866.530476] Key type .llcrypt registered [ 7866.696675] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 7867.560677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7867.628451] alg: No test for adler32 (adler32-zlib) [ 7868.637021] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 7868.795044] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 7870.432824] Key type lgssc registered [ 7871.561861] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7878.701804] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7878.737967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7880.458674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7883.435183] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7885.792746] LustreError: 285100:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7885.823218] LustreError: 285100:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7889.891306] LustreError: 285100:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7890.877027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7891.117269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7893.683433] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7897.636933] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7900.539754] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7906.291880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7906.617421] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7911.808658] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 7916.557243] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7925.043300] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7935.466182] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7935.468820] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7935.472660] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7935.476784] Lustre: Skipped 1 previous similar message [ 7935.483258] Lustre: Skipped 1 previous similar message [ 7940.582875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7940.594655] Lustre: Skipped 1 previous similar message [ 7941.194358] LustreError: 287525:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7941.318855] Lustre: server umount lustre-OST0000 complete [ 7945.701770] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7945.707218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7945.707812] Lustre: Skipped 1 previous similar message [ 7945.717511] Lustre: Skipped 1 previous similar message [ 7950.817263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7950.822152] Lustre: Skipped 1 previous similar message [ 7951.348736] LustreError: 287727:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7951.352715] LustreError: 287727:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7951.497544] Lustre: server umount lustre-MDT0000 complete [ 7955.562067] LustreError: 285831:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763348152 with bad export cookie 366289273052243465 [ 7955.574584] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7955.579578] LustreError: 285831:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7955.703286] LustreError: 287929:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7955.706675] LustreError: 287929:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7956.113090] Lustre: server umount lustre-MDT0001 complete [ 7967.483122] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 7969.737870] Key type lgssc unregistered [ 7969.979970] LNet: 288461:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7969.988962] LNetError: 288461:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7971.046966] LNet: Removed LNI 192.168.202.127@tcp [ 7971.655260] Key type .llcrypt unregistered [ 7971.656828] Key type ._llcrypt unregistered [ 7981.003148] Key type ._llcrypt registered [ 7981.004584] Key type .llcrypt registered [ 7981.121312] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 7995.286596] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 8007.252121] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 8008.025111] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8008.152035] alg: No test for adler32 (adler32-zlib) [ 8009.403440] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 8009.654350] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 8011.322538] Key type lgssc registered [ 8012.616402] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8021.443033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8027.586913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8033.451794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8039.753539] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8045.939695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8046.014902] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8046.034379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8047.219125] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8047.248189] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8047.300960] Lustre: lustre-MDT0000: new disk, initializing [ 8047.372825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8047.386798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8050.673497] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8061.127897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8061.207505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8061.271209] Lustre: 291951:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8061.327923] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8061.331436] Lustre: Skipped 1 previous similar message [ 8061.449279] Lustre: lustre-MDT0001: new disk, initializing [ 8061.556728] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8061.590558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8061.608965] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8065.590502] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8073.205490] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8076.425848] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8082.586878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8082.667286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8082.828851] Lustre: lustre-OST0000: new disk, initializing [ 8082.834909] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8082.906936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8088.165500] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8089.673915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8089.685238] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8089.775978] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8094.677100] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8098.266788] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8098.483076] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8102.280873] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8102.509340] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8105.440590] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8105.443362] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8105.466773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8110.529389] LustreError: 294889:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8110.784881] Lustre: server umount lustre-OST0000 complete [ 8115.683654] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8115.686908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8115.709513] Lustre: Skipped 2 previous similar messages [ 8115.713203] Lustre: Skipped 1 previous similar message [ 8120.557739] LustreError: 295090:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8120.572662] LustreError: 295090:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8120.694869] Lustre: server umount lustre-MDT0000 complete [ 8120.800652] LustreError: 291962:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8120.829590] LustreError: 291962:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 8124.169106] LustreError: 291942:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763348321 with bad export cookie 16663244090845057833 [ 8124.176315] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8124.189709] LustreError: 291942:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8124.322207] LustreError: 295291:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8124.328586] LustreError: 295291:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8124.575658] Lustre: server umount lustre-MDT0001 complete [ 8132.346831] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 21:58:48 (1763348328) [ 8137.679652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8137.983638] LustreError: 295963:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8138.057421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8141.056807] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8143.328788] LustreError: 295963:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8143.336923] LustreError: 295963:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8147.240144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8147.654600] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8150.670952] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8155.220890] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8158.682897] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8164.728730] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8165.043954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8169.743672] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8175.382801] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8194.024889] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8194.029935] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8194.032251] Lustre: Skipped 1 previous similar message [ 8194.036139] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8194.036147] LustreError: Skipped 1 previous similar message [ 8194.061207] Lustre: Skipped 2 previous similar messages [ 8199.142638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8199.153649] Lustre: Skipped 1 previous similar message [ 8199.736771] LustreError: 298232:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8199.749150] LustreError: 298232:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8199.919991] Lustre: server umount lustre-OST0000 complete [ 8203.747116] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8203.750502] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8203.766761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8209.409539] LustreError: 298435:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8209.412115] LustreError: 298435:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8211.517065] Lustre: server umount lustre-MDT0000 complete [ 8214.497206] LustreError: 295958:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8214.511954] LustreError: 295958:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8214.601553] LustreError: 295944:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763348411 with bad export cookie 16663244090845058715 [ 8214.607940] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8214.611477] LustreError: 295944:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8214.944334] Lustre: server umount lustre-MDT0001 complete [ 8222.201329] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 8224.249587] Key type lgssc unregistered [ 8224.507779] LNet: 299170:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8224.521177] LNetError: 299170:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8224.535309] LNet: Removed LNI 192.168.202.127@tcp [ 8225.454476] Key type .llcrypt unregistered [ 8225.456815] Key type ._llcrypt unregistered [ 8248.634832] Key type ._llcrypt registered [ 8248.636984] Key type .llcrypt registered [ 8248.767450] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 8261.723342] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 8262.456177] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8262.542768] alg: No test for adler32 (adler32-zlib) [ 8263.682848] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 8263.899340] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 8265.623204] Key type lgssc registered [ 8266.799286] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8274.588621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8280.349980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8285.419317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8291.418137] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8296.249075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8301.173362] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8307.409465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8307.446975] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8307.464705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8308.675058] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8308.703467] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8308.749510] Lustre: lustre-MDT0000: new disk, initializing [ 8308.815529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8308.834818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8312.374645] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8321.724677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8321.777124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8321.830938] Lustre: 302991:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8321.906851] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8321.921152] Lustre: Skipped 1 previous similar message [ 8322.056336] Lustre: lustre-MDT0001: new disk, initializing [ 8322.178213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8322.213580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8322.231511] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8325.333393] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8330.268983] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8332.464483] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8336.793858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8336.851815] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8337.034190] Lustre: lustre-OST2710: new disk, initializing [ 8337.040885] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8337.116671] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8341.651444] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8346.225529] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8346.236268] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8346.320229] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8348.496866] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 8355.221202] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8355.294702] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8355.405406] Lustre: lustre-OST03e8: new disk, initializing [ 8355.410932] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8355.519246] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8360.679407] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8365.202641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8365.206491] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8365.373728] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8365.597367] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 8371.187344] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8371.335131] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8375.395587] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8375.548387] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8380.056976] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8380.230482] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8384.000797] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8384.232864] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8388.576167] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8388.580975] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8388.594505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8391.139692] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8391.141479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8391.149779] Lustre: Skipped 1 previous similar message [ 8391.156648] Lustre: Skipped 1 previous similar message [ 8394.062175] LustreError: 307333:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8394.152775] Lustre: server umount lustre-MDT0000 complete [ 8397.825149] LustreError: 302984:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763348595 with bad export cookie 5320319581391109697 [ 8397.836491] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8397.845251] LustreError: 302984:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8397.980840] LustreError: 307534:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8397.989596] LustreError: 307534:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8398.210663] Lustre: server umount lustre-MDT0001 complete [ 8412.556227] LustreError: 307736:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8412.564179] LustreError: 307736:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8412.599406] Lustre: server umount lustre-OST2710 complete [ 8424.829059] LustreError: 307938:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 8424.832444] LustreError: 307938:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8424.936573] Lustre: server umount lustre-OST03e8 complete [ 8439.947910] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 8448.653897] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 8456.876045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8462.213381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8468.513615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8474.470342] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8480.812353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8480.892706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8481.133892] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8481.169619] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8481.232186] Lustre: lustre-MDT0000: new disk, initializing [ 8481.300356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8481.317624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8484.333601] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8492.606820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8492.655830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8492.710135] Lustre: 310736:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8492.759887] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8492.762319] Lustre: Skipped 1 previous similar message [ 8492.832488] Lustre: lustre-MDT0001: new disk, initializing [ 8492.909203] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8492.953197] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8492.973664] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8495.686927] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8501.394428] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8503.937166] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8508.262287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8508.327303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8512.758574] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8518.193651] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8518.198241] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8518.337644] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8519.912252] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8523.431107] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8523.583833] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8527.087408] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8527.280837] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8528.865045] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8528.869867] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8528.881303] LustreError: Skipped 1 previous similar message [ 8528.889902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8528.897017] Lustre: Skipped 2 previous similar messages [ 8528.903384] Lustre: Skipped 2 previous similar messages [ 8533.988111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8533.995083] Lustre: Skipped 2 previous similar messages [ 8539.106082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8539.115630] Lustre: Skipped 1 previous similar message [ 8543.199137] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8543.231115] LustreError: 313674:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8543.237572] LustreError: 313674:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8543.356985] Lustre: server umount lustre-OST0000 complete [ 8549.344968] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8549.347281] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8549.347682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8549.351307] Lustre: Skipped 1 previous similar message [ 8552.696744] LustreError: 313878:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8552.701523] LustreError: 313878:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8552.775362] Lustre: server umount lustre-MDT0000 complete [ 8554.465964] LustreError: 310747:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8554.482868] LustreError: 310747:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 8555.511127] LustreError: 312481:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763348752 with bad export cookie 5320319581391111377 [ 8555.517077] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8555.523231] LustreError: 312481:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 8555.829931] Lustre: server umount lustre-MDT0001 complete [ 8563.014553] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 22:05:58 (1763348758) [ 8564.367907] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8565.904583] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 22:06:01 (1763348761) [ 8568.193170] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 8574.222949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8577.046588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8579.871139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8582.651089] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8583.501175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8588.554066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8588.731624] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8588.747468] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8588.865678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8588.869559] Lustre: Skipped 1 previous similar message [ 8591.450638] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8598.347134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8598.412240] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8598.436910] Lustre: 316210:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8601.926726] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8607.041363] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8610.578299] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8618.216570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8618.308595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8618.432538] Lustre: Denying initial registration attempt from nid 192.168.202.127@tcp, specified as failover [ 8618.439304] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8618.459671] LustreError: 317826:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -99 [ 8618.464026] LustreError: 317826:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 8618.470347] LustreError: 317826:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 8618.521961] Lustre: server umount lustre-OST0000 complete [ 8618.525714] LustreError: 317826:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8624.611974] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8624.618201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8624.632422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8624.635806] Lustre: Skipped 1 previous similar message [ 8630.337077] LustreError: 318195:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 8630.340778] LustreError: 318195:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 8630.384820] Lustre: server umount lustre-MDT0000 complete [ 8632.289720] LustreError: 318065:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8632.309797] LustreError: 318065:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8633.497443] LustreError: 318196:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763348830 with bad export cookie 5320319581391112259 [ 8633.501297] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8633.503480] LustreError: 318196:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8633.754976] Lustre: server umount lustre-MDT0001 complete [ 8643.215257] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 8646.128643] Key type lgssc unregistered [ 8646.423713] LNet: 318930:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8646.439078] LNetError: 318930:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8646.463769] LNet: Removed LNI 192.168.202.127@tcp [ 8647.266214] Key type .llcrypt unregistered [ 8647.269505] Key type ._llcrypt unregistered [ 8658.677980] Key type ._llcrypt registered [ 8658.696315] Key type .llcrypt registered [ 8658.812745] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 8659.887068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8659.907480] alg: No test for adler32 (adler32-zlib) [ 8661.502856] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 8661.729266] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 8663.400431] Key type lgssc registered [ 8664.452961] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8670.877427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8674.953658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8679.330846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8683.161982] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8684.160393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8700.443661] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 8708.125309] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8708.151601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8709.419485] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8709.440969] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8709.628467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8713.579826] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8723.382905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8723.491280] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8723.518357] Lustre: 321690:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8723.753930] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8728.002707] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8734.596828] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8738.414571] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8745.012453] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8745.119462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8745.445122] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8745.619556] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8750.492903] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8755.347217] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8761.315156] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8761.316083] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8761.327263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8761.335638] Lustre: Skipped 1 previous similar message [ 8763.386282] LustreError: 323975:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8763.488396] Lustre: server umount lustre-OST0000 complete [ 8770.020517] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8770.044667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8770.086272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8770.088681] Lustre: Skipped 1 previous similar message [ 8771.552782] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8771.561355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8771.564192] Lustre: Skipped 1 previous similar message [ 8776.672333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8776.675078] Lustre: Skipped 1 previous similar message [ 8781.282846] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8781.377032] LustreError: 324176:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 8781.380021] LustreError: 324176:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8781.472548] Lustre: server umount lustre-MDT0000 complete [ 8784.818761] LustreError: 321681:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763348982 with bad export cookie 8350332028274657838 [ 8784.823347] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8784.832141] LustreError: 321681:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8784.947782] LustreError: 324378:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8784.951664] LustreError: 324378:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8785.274741] Lustre: server umount lustre-MDT0001 complete [ 8795.248681] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 8797.746583] Key type lgssc unregistered [ 8798.003316] LNet: 324911:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8798.009448] LNetError: 324911:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8798.023053] LNet: Removed LNI 192.168.202.127@tcp [ 8798.757568] Key type .llcrypt unregistered [ 8798.759335] Key type ._llcrypt unregistered [ 8820.219690] Key type ._llcrypt registered [ 8820.221236] Key type .llcrypt registered [ 8820.304875] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 8821.111532] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8821.120122] alg: No test for adler32 (adler32-zlib) [ 8822.196223] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 8822.438849] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 8824.135151] Key type lgssc registered [ 8825.353132] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8833.383472] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8833.406535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8835.001874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8838.666953] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8845.282129] LustreError: 326308:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8845.314195] LustreError: 326308:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8847.173612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8847.621481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8851.487399] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8856.517950] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8860.566408] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8868.227995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8868.608342] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8873.933756] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8878.756779] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8893.922397] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8893.924942] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8893.932368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8894.652542] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.27@tcp (stopping) [ 8895.463210] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8897.540282] LustreError: 328582:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8897.642037] Lustre: server umount lustre-MDT0000 complete [ 8899.753807] LustreError: 326302:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8901.383935] LustreError: 326289:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763349098 with bad export cookie 18332021581194072576 [ 8901.391544] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8901.411635] LustreError: 326289:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8904.876100] LustreError: 326304:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8904.910807] LustreError: 326304:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8904.928177] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.27@tcp (stopping) [ 8904.937129] Lustre: Skipped 3 previous similar messages [ 8905.697931] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8905.716544] Lustre: Skipped 1 previous similar message [ 8907.665814] LustreError: 328784:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8907.682674] LustreError: 328784:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8907.958355] Lustre: server umount lustre-MDT0001 complete [ 8914.959908] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8925.623964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8926.634251] LustreError: 327919:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8926.642644] LustreError: 327919:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8926.801634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8926.898410] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:257) [ 8930.852920] LustreError: 329591:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8930.867827] LustreError: 329591:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8931.375424] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8936.937626] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8939.304360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8939.649421] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8943.751917] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 8945.125459] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8949.330001] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8951.384433] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8955.620659] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8957.523227] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8962.538237] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8962.541793] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8962.547357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8962.549809] Lustre: Skipped 2 previous similar messages [ 8965.089324] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8968.644110] LustreError: 331089:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8968.655967] LustreError: 331089:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8968.800098] Lustre: server umount lustre-OST0000 complete [ 8975.328398] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8975.336713] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8975.345861] Lustre: Skipped 1 previous similar message [ 8975.350136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8975.359275] Lustre: Skipped 2 previous similar messages [ 8978.504802] LustreError: 331291:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8978.507744] LustreError: 331291:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8978.597386] Lustre: server umount lustre-MDT0000 complete [ 8980.962825] LustreError: 329596:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8980.976536] LustreError: 329596:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 8982.050935] LustreError: 326289:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763349179 with bad export cookie 18332021581194080227 [ 8982.053329] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8982.105167] LustreError: 326289:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8982.547091] Lustre: server umount lustre-MDT0001 complete [ 8990.771298] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 8993.303492] Key type lgssc unregistered [ 8993.626285] LNet: 332025:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8993.630703] LNetError: 332025:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8993.644562] LNet: Removed LNI 192.168.202.127@tcp [ 8994.490228] Key type .llcrypt unregistered [ 8994.491773] Key type ._llcrypt unregistered [ 9013.184906] Key type ._llcrypt registered [ 9013.187491] Key type .llcrypt registered [ 9013.275903] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 9013.894393] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9013.920857] alg: No test for adler32 (adler32-zlib) [ 9015.057932] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 9015.314396] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 9016.999140] Key type lgssc registered [ 9018.202985] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9026.110781] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9026.127601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9027.738483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9031.632652] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9037.284528] LustreError: 333400:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9037.291981] LustreError: 333400:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9039.956792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9040.281551] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9043.949141] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9049.265617] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9052.665287] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9055.713973] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9055.723502] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9055.734516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9057.760577] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9057.764169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9061.371377] LustreError: 334793:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9061.493285] Lustre: server umount lustre-MDT0000 complete [ 9062.884919] LustreError: 333395:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9062.899157] LustreError: 333395:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9064.409532] LustreError: 333379:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763349261 with bad export cookie 3884859929162085756 [ 9064.418907] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9064.547928] LustreError: 334995:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 9064.557469] LustreError: 334995:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9064.727922] Lustre: server umount lustre-MDT0001 complete [ 9078.247982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9078.660721] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9078.695407] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9078.945610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9082.264783] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9090.279373] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9090.432322] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9090.469693] Lustre: 335764:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9090.712490] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9094.132480] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9099.645412] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9103.310778] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9109.570640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9109.682660] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9109.691171] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9109.696436] LustreError: 337424:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 9109.700859] LustreError: 337424:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 9109.704215] LustreError: 337424:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 9109.737931] Lustre: server umount lustre-OST0000 complete [ 9109.740318] LustreError: 337424:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9115.585324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9115.723586] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9115.888314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9120.043148] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9124.424937] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:289) [ 9124.609167] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9130.978621] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9131.030818] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9131.074576] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9131.135353] Lustre: lustre-OST0001: new disk, initializing [ 9131.146510] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9131.231941] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9135.583322] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9136.632774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9136.649539] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9136.686881] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9139.958366] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9146.851276] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9146.853038] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9146.855415] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9146.859284] Lustre: Skipped 1 previous similar message [ 9146.865536] Lustre: Skipped 1 previous similar message [ 9148.551304] LustreError: 339549:0:(obd_class.h:479:obd_check_dev()) Device 26 not setup [ 9148.554191] LustreError: 339549:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 9148.598258] Lustre: server umount lustre-OST0001 complete [ 9151.974403] LustreError: 338430:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9151.977147] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9151.980853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9151.980866] Lustre: Skipped 1 previous similar message [ 9151.997121] LustreError: 338430:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9152.029335] Lustre: Skipped 1 previous similar message [ 9157.092336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9157.094107] LustreError: 337891:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9157.098055] Lustre: Skipped 2 previous similar messages [ 9157.783269] LustreError: 339751:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9157.797078] LustreError: 339751:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9158.025462] Lustre: server umount lustre-OST0000 complete [ 9162.209235] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9162.223764] Lustre: Skipped 1 previous similar message [ 9167.328382] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9167.337086] Lustre: Skipped 3 previous similar messages [ 9167.411538] LustreError: 339952:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 9167.414248] LustreError: 339952:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9167.536437] Lustre: server umount lustre-MDT0000 complete [ 9171.169393] LustreError: 337908:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763349368 with bad export cookie 3884859929162086309 [ 9171.176587] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9171.181756] LustreError: 337908:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 9171.519115] Lustre: server umount lustre-MDT0001 complete [ 9181.419784] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 9184.052451] Key type lgssc unregistered [ 9184.269941] LNet: 340687:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9184.277084] LNetError: 340687:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9184.289167] LNet: Removed LNI 192.168.202.127@tcp [ 9185.054084] Key type .llcrypt unregistered [ 9185.055976] Key type ._llcrypt unregistered [ 9215.844041] Key type ._llcrypt registered [ 9215.845794] Key type .llcrypt registered [ 9215.957703] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 9230.157357] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 9230.952598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9230.967310] alg: No test for adler32 (adler32-zlib) [ 9232.185045] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 9232.440815] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 9234.095694] Key type lgssc registered [ 9235.282239] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9243.272909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9249.430110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9254.385647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9259.784385] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9265.094860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9265.130172] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9265.144568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9266.341349] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9266.368866] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9266.419891] Lustre: lustre-MDT0000: new disk, initializing [ 9266.518454] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9266.530745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9269.272676] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9277.452178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9277.508622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9277.584961] Lustre: 344639:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9277.632869] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9277.636376] Lustre: Skipped 1 previous similar message [ 9277.716849] Lustre: lustre-MDT0001: new disk, initializing [ 9277.783183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9277.814338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9277.825399] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9280.675122] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9286.434326] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9289.280425] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9294.713503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9294.792659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9295.013835] Lustre: lustre-OST0000: new disk, initializing [ 9295.024498] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9295.100773] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9295.424393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9295.432590] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9295.503428] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9299.546303] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9306.004545] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9309.736691] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9309.908615] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9313.076641] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9313.228814] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9316.327167] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9316.327617] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9316.337422] LustreError: Skipped 1 previous similar message [ 9316.338336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9316.346144] Lustre: Skipped 1 previous similar message [ 9321.443895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9321.447064] Lustre: Skipped 2 previous similar messages [ 9326.567936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9326.575623] Lustre: Skipped 1 previous similar message [ 9329.122398] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9329.186895] LustreError: 347575:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9329.310572] Lustre: server umount lustre-OST0000 complete [ 9334.242466] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9334.247577] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9334.281936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9336.806418] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9339.454763] LustreError: 347779:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 9339.458442] LustreError: 347779:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9339.574860] Lustre: server umount lustre-MDT0000 complete [ 9343.767257] LustreError: 344631:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763349540 with bad export cookie 11741335290413900186 [ 9343.773930] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9343.790535] LustreError: 344631:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9343.986304] LustreError: 347980:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9344.006936] LustreError: 347980:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9344.265775] Lustre: server umount lustre-MDT0001 complete [ 9353.629525] Lustre: DEBUG MARKER: == conf-sanity test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 22:19:09 (1763349549) [ 9359.259271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9365.036687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9385.507392] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [ 9393.923352] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 9402.845460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9409.128811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9415.741411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9422.192465] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9430.087627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9430.169700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9430.337612] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9430.365032] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9430.423709] Lustre: lustre-MDT0000: new disk, initializing [ 9430.495847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9430.511754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9434.119751] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9443.900606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9443.977107] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9444.077948] Lustre: 351759:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9444.157397] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9444.164663] Lustre: Skipped 1 previous similar message [ 9444.245705] Lustre: lustre-MDT0001: new disk, initializing [ 9444.320844] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9444.359756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9444.367461] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9448.225387] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9456.044737] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9460.105942] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9467.019467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9467.133664] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9467.463250] Lustre: lustre-OST0000: new disk, initializing [ 9467.471169] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9467.598207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9473.486291] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9473.611781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9473.627917] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9473.798782] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9481.124391] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9485.170045] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9485.373372] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9489.215976] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9489.447529] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9494.501735] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9494.504796] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9494.504928] LustreError: Skipped 1 previous similar message [ 9494.506250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9494.506256] Lustre: Skipped 2 previous similar messages [ 9494.538270] Lustre: Skipped 1 previous similar message [ 9497.469724] LustreError: 354696:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9497.481978] LustreError: 354696:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 9497.632496] Lustre: server umount lustre-OST0000 complete [ 9504.737676] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9504.741710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9504.749748] Lustre: Skipped 1 previous similar message [ 9504.756962] Lustre: Skipped 1 previous similar message [ 9507.201747] LustreError: 354899:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 9507.217403] LustreError: 354899:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9507.350348] Lustre: server umount lustre-MDT0000 complete [ 9509.857535] LustreError: 352891:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9509.878348] LustreError: 352891:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 9510.568903] LustreError: 351751:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763349707 with bad export cookie 11741335290413901068 [ 9510.578886] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9510.982632] Lustre: server umount lustre-MDT0001 complete [ 9520.598445] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 22:21:56 (1763349716) [ 9528.900673] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 22:22:04 (1763349724) [ 9538.650334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9539.118224] LustreError: 356115:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9539.188756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9542.908142] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9544.680055] LustreError: 356115:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9549.794637] LustreError: 356115:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9551.763896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9556.360406] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9561.617401] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9565.930955] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9573.608772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9574.230088] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9574.233153] Lustre: Skipped 1 previous similar message [ 9580.019734] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9585.945529] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9593.346991] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9595.213745] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9597.113488] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9598.801388] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9600.800053] Lustre: Failing over lustre-MDT0000 [ 9600.997124] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9601.004390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9601.023289] LustreError: 358555:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9601.023305] LustreError: 358555:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 9601.028825] Lustre: Skipped 1 previous similar message [ 9601.051762] Lustre: Skipped 2 previous similar messages [ 9601.161779] Lustre: server umount lustre-MDT0000 complete [ 9603.284552] LustreError: 356111:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9613.483245] LustreError: 356111:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9613.496107] LustreError: 356111:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 9619.362401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9619.453822] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9619.765746] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9624.281795] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9624.743292] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9625.075206] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9625.090945] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 9625.134326] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9631.073717] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9632.738849] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9636.047954] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9638.196056] Lustre: Failing over lustre-MDT0000 [ 9638.382137] LustreError: 359635:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9638.385228] LustreError: 359635:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9638.623734] Lustre: server umount lustre-MDT0000 complete [ 9640.126917] LustreError: 356112:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9640.133813] LustreError: 356112:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 9640.421554] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9640.425711] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9640.452645] Lustre: Skipped 3 previous similar messages [ 9656.735152] Lustre: 342849:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1763349837/real 1763349837] req@000000006242b32b x1849005890985344/t0(0) o400->MGC192.168.202.127@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1763349853 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9656.751895] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9657.120143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9666.017146] LustreError: 342848:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000e0575077 x1849005890990976/t0(0) o250->MGC192.168.202.127@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9666.327432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9666.330648] Lustre: Skipped 1 previous similar message [ 9666.382631] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9666.733245] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9670.322845] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9671.666118] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9671.674631] Lustre: Skipped 2 previous similar messages [ 9671.719225] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 9671.826544] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9678.528535] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9680.371246] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9687.014586] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9687.021216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9692.536763] LustreError: 360679:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9692.539338] LustreError: 360679:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9692.712167] Lustre: server umount lustre-MDT0000 complete [ 9696.186244] LustreError: 357737:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763349893 with bad export cookie 11741335290413903763 [ 9696.187679] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9696.197300] LustreError: 357737:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9696.613666] Lustre: server umount lustre-MDT0001 complete [ 9710.741460] Lustre: server umount lustre-OST0000 complete [ 9725.457210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9726.010392] LustreError: 361910:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9726.033619] LustreError: 361910:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 26 previous similar messages [ 9730.083742] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9738.921071] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9742.888162] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9748.725460] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9752.047515] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9758.771776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9764.824607] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9770.567666] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9771.441576] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9779.680485] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9779.687680] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9779.716457] Lustre: Skipped 2 previous similar messages [ 9779.737162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9779.741626] Lustre: Skipped 5 previous similar messages [ 9785.663932] LustreError: 364154:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9785.673858] LustreError: 364154:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 9785.821464] Lustre: server umount lustre-OST0000 complete [ 9790.436375] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9790.449966] LustreError: Skipped 1 previous similar message [ 9795.816147] Lustre: server umount lustre-MDT0000 complete [ 9797.091035] LustreError: 361910:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9797.117071] LustreError: 361910:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9799.109875] LustreError: 361892:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763349996 with bad export cookie 11741335290413904498 [ 9799.122363] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9799.125751] LustreError: 361892:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9799.473033] Lustre: server umount lustre-MDT0001 complete [ 9809.049223] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [ 9812.024660] Key type lgssc unregistered [ 9812.428453] LNet: 365093:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9812.436344] LNetError: 365093:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9812.456605] LNet: Removed LNI 192.168.202.127@tcp [ 9813.213745] Key type .llcrypt unregistered [ 9813.215690] Key type ._llcrypt unregistered [ 9844.622441] Key type ._llcrypt registered [ 9844.624267] Key type .llcrypt registered [ 9844.716633] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [ 9845.508040] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9845.535588] alg: No test for adler32 (adler32-zlib) [ 9847.059924] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [ 9847.498801] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [ 9849.348098] Key type lgssc registered [ 9850.559690] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9860.107628] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9860.140642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9862.017837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9866.392931] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9872.355264] LustreError: 366591:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9872.365202] LustreError: 366591:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9875.732294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9876.251805] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9880.423585] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9886.150571] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9890.135186] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9897.262364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9897.585145] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9903.108683] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9906.821611] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9909.544923] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9922.532432] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9922.535195] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9922.540104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9922.550808] Lustre: Skipped 1 previous similar message [ 9925.877226] LustreError: 368840:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9925.979796] Lustre: server umount lustre-MDT0000 complete [ 9927.656452] LustreError: 366585:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9927.679746] LustreError: 366585:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9929.988285] LustreError: 366572:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350127 with bad export cookie 9923250080018958596 [ 9929.992201] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9930.004807] LustreError: 366572:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9930.176113] LustreError: 369041:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 9930.185021] LustreError: 369041:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9930.416786] Lustre: server umount lustre-MDT0001 complete [ 9944.458552] LustreError: 369242:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 9944.460690] LustreError: 369242:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9944.591479] Lustre: server umount lustre-OST0000 complete [ 9957.191386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9957.904465] LustreError: 369973:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9957.955955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9961.983650] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9962.983213] LustreError: 369973:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9968.099389] LustreError: 369973:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9970.634176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9970.989985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9975.475471] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [ 9980.651656] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9984.048777] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9990.620685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9990.918427] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9996.087279] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10001.515702] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10003.196714] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10018.787500] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10018.798697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10018.800726] Lustre: Skipped 2 previous similar messages [10018.820158] Lustre: Skipped 4 previous similar messages [10023.911643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10023.926672] Lustre: Skipped 2 previous similar messages [10024.304521] LustreError: 372323:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10024.306987] LustreError: 372323:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10024.554860] Lustre: server umount lustre-MDT0000 complete [10028.766106] LustreError: 372236:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350225 with bad export cookie 9923250080018959709 [10028.768500] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10028.779970] LustreError: 372236:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [10029.019622] LustreError: 372524:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [10029.025043] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10029.034807] LustreError: 372524:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10029.048236] Lustre: Skipped 1 previous similar message [10029.051029] LustreError: 371592:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10029.062148] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10029.255725] Lustre: server umount lustre-MDT0001 complete [10036.107312] Lustre: server umount lustre-OST0000 complete [10054.775683] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10055.091967] LustreError: 373553:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10055.160065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10059.766684] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10069.864956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10074.262512] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10079.635876] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10083.688753] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10090.883190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10091.145321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10091.147612] Lustre: Skipped 1 previous similar message [10096.977022] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10100.405656] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10102.508295] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10110.950059] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10110.953321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10110.981699] Lustre: Skipped 1 previous similar message [10121.185075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10121.188027] Lustre: Skipped 3 previous similar messages [10121.188450] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [10121.243141] LustreError: 375797:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10121.254517] LustreError: 375797:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [10121.368508] Lustre: server umount lustre-OST0000 complete [10126.306786] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10131.227409] Lustre: server umount lustre-MDT0000 complete [10131.425619] LustreError: 374578:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10131.458246] LustreError: 374578:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10134.984090] LustreError: 373534:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350332 with bad export cookie 9923250080018960731 [10134.995805] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10135.317427] Lustre: server umount lustre-MDT0001 complete [10144.308442] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [10147.088680] Key type lgssc unregistered [10147.355338] LNet: 376736:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10147.360696] LNetError: 376736:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10147.378365] LNet: Removed LNI 192.168.202.127@tcp [10148.219918] Key type .llcrypt unregistered [10148.221705] Key type ._llcrypt unregistered [10176.613535] Key type ._llcrypt registered [10176.617394] Key type .llcrypt registered [10176.767062] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [10178.033056] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10178.121786] alg: No test for adler32 (adler32-zlib) [10179.270404] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [10179.520214] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [10181.191234] Key type lgssc registered [10182.325666] Lustre: Echo OBD driver; http://www.lustre.org/ [10188.690667] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10188.705626] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10188.710641] LustreError: 378155:0:(osd_handler.c:8577:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10188.724944] LustreError: 378155:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000-osd failed (-22) [10188.728975] LustreError: 378155:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10188.734247] LustreError: 378155:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10188.739921] LustreError: 378155:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [10194.682143] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [10194.686403] LustreError: 378553:0:(osd_handler.c:8577:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10194.703294] LustreError: 378553:0:(obd_config.c:783:class_setup()) setup lustre-OST0000-osd failed (-22) [10194.707400] LustreError: 378553:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10194.715954] LustreError: 378553:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10194.723156] LustreError: 378553:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [10211.563901] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [10214.314335] Key type lgssc unregistered [10214.613705] LNet: 379540:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10214.625520] LNetError: 379540:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10214.655328] LNet: Removed LNI 192.168.202.127@tcp [10215.378859] Key type .llcrypt unregistered [10215.380553] Key type ._llcrypt unregistered [10231.687877] Key type ._llcrypt registered [10231.689930] Key type .llcrypt registered [10231.797094] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [10244.592189] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [10245.412446] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10245.496755] alg: No test for adler32 (adler32-zlib) [10246.526086] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [10246.854342] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [10248.535247] Key type lgssc registered [10249.612816] Lustre: Echo OBD driver; http://www.lustre.org/ [10257.945137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10264.048818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10269.609559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10275.726166] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10282.571072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10282.637965] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10282.657918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10283.850514] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10283.879504] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10283.945509] Lustre: lustre-MDT0000: new disk, initializing [10284.039430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10284.059397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10287.985371] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10297.323967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10297.419449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10297.477782] Lustre: 382681:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10297.520984] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10297.524105] Lustre: Skipped 1 previous similar message [10297.577731] Lustre: lustre-MDT0001: new disk, initializing [10297.670776] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10297.753187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10297.761801] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10301.108553] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10308.263239] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10312.323253] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10318.280503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10318.338088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10318.526455] Lustre: lustre-OST0000: new disk, initializing [10318.531802] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10318.629229] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10323.350859] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10328.120238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10328.131403] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10328.241771] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10329.866537] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10333.560109] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10333.788445] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10337.048787] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10337.263362] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10343.907589] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10343.911923] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10343.943567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10343.959282] Lustre: Skipped 1 previous similar message [10349.041793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10353.119202] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10353.162125] LustreError: 385622:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [10353.319578] Lustre: server umount lustre-OST0000 complete [10359.270214] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10359.274323] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10359.274927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10359.274931] Lustre: Skipped 1 previous similar message [10359.290304] Lustre: Skipped 2 previous similar messages [10363.634045] LustreError: 385825:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [10363.641014] LustreError: 385825:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10363.894211] Lustre: server umount lustre-MDT0000 complete [10367.314164] LustreError: 382674:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350564 with bad export cookie 4376056151157738342 [10367.322460] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10367.325596] LustreError: 382674:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [10367.473661] LustreError: 386026:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10367.475988] LustreError: 386026:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10367.769497] Lustre: server umount lustre-MDT0001 complete [10375.756620] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10377.125755] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 22:36:12 (1763350572) [10382.424345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10382.747888] LustreError: 386745:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10382.757525] LustreError: 386745:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10382.822776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10386.168576] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10387.938899] LustreError: 386745:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10392.034358] LustreError: 386745:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10392.906535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10393.253056] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10396.713163] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10401.860985] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10405.785396] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10412.059873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10412.395427] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10417.096292] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10421.367665] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10427.924297] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10427.983598] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10428.080796] Lustre: lustre-OST0001: new disk, initializing [10428.088276] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10428.651191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10428.673800] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10428.775341] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10433.038397] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10440.401828] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [10444.259497] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10444.265150] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10444.273955] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10444.277123] Lustre: Skipped 1 previous similar message [10444.288317] Lustre: Skipped 1 previous similar message [10449.381679] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10449.391304] Lustre: Skipped 3 previous similar messages [10450.174902] LustreError: 389945:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [10450.196040] LustreError: 389945:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [10450.594668] Lustre: server umount lustre-OST0001 complete [10452.148375] LustreError: 388345:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.202.27@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10459.619471] LustreError: 388797:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10459.626730] LustreError: 388797:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10468.832684] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10468.847894] LustreError: Skipped 1 previous similar message [10468.852650] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10468.877744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10469.866786] LustreError: 388346:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10469.904442] LustreError: 388346:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10481.119472] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10481.146582] LustreError: 390170:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10481.154589] LustreError: 390170:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10481.246633] Lustre: server umount lustre-OST0000 complete [10485.217945] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10485.228975] Lustre: Skipped 1 previous similar message [10485.232576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10485.237996] Lustre: Skipped 7 previous similar messages [10490.749263] LustreError: 390376:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [10490.763658] LustreError: 390376:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10490.883494] Lustre: server umount lustre-MDT0000 complete [10494.304163] LustreError: 386724:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350691 with bad export cookie 4376056151157739217 [10494.306793] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10494.319214] LustreError: 386724:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [10500.644872] Lustre: server umount lustre-MDT0001 complete [10510.528331] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [10513.445532] Key type lgssc unregistered [10513.759938] LNet: 391110:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10513.769426] LNetError: 391110:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10514.795462] LNet: Removed LNI 192.168.202.127@tcp [10515.549873] Key type .llcrypt unregistered [10515.551604] Key type ._llcrypt unregistered [10542.688059] Key type ._llcrypt registered [10542.689481] Key type .llcrypt registered [10542.826261] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [10556.122916] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [10557.008271] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10557.051468] alg: No test for adler32 (adler32-zlib) [10558.128360] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [10558.306085] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [10559.975189] Key type lgssc registered [10560.805796] Lustre: Echo OBD driver; http://www.lustre.org/ [10567.429701] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10573.914313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10580.752149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10586.436824] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10593.213275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10593.252319] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10593.265559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10594.492982] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10594.541741] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10594.657283] Lustre: lustre-MDT0000: new disk, initializing [10594.740829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10594.760729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10598.488966] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10607.216567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10607.262498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10607.314287] Lustre: 395067:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10607.357807] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10607.362367] Lustre: Skipped 1 previous similar message [10607.414426] Lustre: lustre-MDT0001: new disk, initializing [10607.499418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10607.536918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10607.549313] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10610.270775] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10617.226518] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10620.495334] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10626.431282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10626.524292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10626.845203] Lustre: lustre-OST0000: new disk, initializing [10626.851283] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10626.931084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10632.103868] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10635.288736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10635.294399] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10635.370518] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10637.424572] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10640.565913] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10640.723860] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10643.716723] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10643.931494] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10650.594831] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10650.596675] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10650.612767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10651.647150] LustreError: 398009:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [10651.790239] Lustre: server umount lustre-OST0000 complete [10655.713815] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10655.717987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10655.743797] Lustre: Skipped 2 previous similar messages [10655.759251] Lustre: Skipped 2 previous similar messages [10660.837905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10660.844806] Lustre: Skipped 1 previous similar message [10661.291475] LustreError: 398211:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [10661.305715] LustreError: 398211:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10661.534841] Lustre: server umount lustre-MDT0000 complete [10665.118623] LustreError: 395060:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350862 with bad export cookie 11832063040862341197 [10665.120302] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10665.161196] LustreError: 395060:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10665.403745] LustreError: 398413:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10665.408612] LustreError: 398413:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10665.599874] Lustre: server umount lustre-MDT0001 complete [10673.388815] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 22:41:09 (1763350869) [10681.526510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10689.463578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10689.756453] Lustre: lustre-MDT0000: new disk, initializing [10689.895838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10693.185565] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10695.136696] LustreError: 399681:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10695.154875] LustreError: 399681:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10700.256762] LustreError: 399681:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10701.556137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10701.849405] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10705.255389] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10709.846159] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10712.610444] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10717.153617] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10717.154178] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10717.169538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10717.174992] Lustre: Skipped 1 previous similar message [10720.495221] LustreError: 401023:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10720.501409] LustreError: 401023:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [10720.587099] Lustre: server umount lustre-MDT0000 complete [10722.272301] LustreError: 399681:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10722.290586] LustreError: 399681:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10723.619574] LustreError: 399660:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763350920 with bad export cookie 11832063040862342072 [10723.632709] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10723.639310] LustreError: 399660:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [10723.863710] Lustre: server umount lustre-MDT0001 complete [10731.990261] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 22:42:07 (1763350927) [10738.209142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10738.538166] LustreError: 401945:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10738.600984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10741.851434] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10743.777546] LustreError: 401945:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10748.882994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10752.601921] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10757.438339] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10761.212618] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10766.638380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10766.858814] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10766.861704] Lustre: Skipped 1 previous similar message [10773.345057] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10778.623143] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10790.267388] Lustre: Permanently deactivating lustre-OST0000 [10790.282584] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [10799.639984] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10801.184812] LustreError: 404638:0:(mgs_llog.c:1938:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [10801.197187] LustreError: 404638:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [10801.206325] LustreError: 404638:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10803.407739] LustreError: 404686:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [10803.414496] LustreError: 404686:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10811.206909] LustreError: 404783:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10811.210165] LustreError: 404783:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [10811.352390] Lustre: server umount lustre-OST0000 complete [10813.696393] LustreError: 404936:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [10813.699703] LustreError: 404936:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10815.969395] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10815.977932] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10815.997464] Lustre: Skipped 1 previous similar message [10816.000847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10820.065852] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10825.188754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10825.198278] Lustre: Skipped 3 previous similar messages [10825.198277] LustreError: 405033:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10825.198292] LustreError: 405033:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10825.274152] Lustre: server umount lustre-MDT0000 complete [10828.086657] LustreError: 401925:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351025 with bad export cookie 11832063040862342632 [10828.091383] LustreError: 401925:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 7 previous similar messages [10828.092169] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10838.681659] Lustre: server umount lustre-MDT0001 complete [10845.439921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10848.876656] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10851.263622] LustreError: 406079:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [10855.925790] Lustre: 406418:0:(mgs_llog.c:1699:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.127@tcp [10857.480522] LustreError: 406517:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [10857.489068] LustreError: 406517:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [10857.571384] Lustre: server umount MGS complete [10866.639133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10866.905851] LustreError: 407146:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10866.919680] LustreError: 407146:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10866.925740] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10866.929145] Lustre: Skipped 1 previous similar message [10866.976987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10870.213708] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10877.558423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10877.769626] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10881.250724] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10885.466222] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10889.127626] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10890.504695] Lustre: Permanently reactivating lustre-OST0000 [10890.517446] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [10890.526805] Lustre: Skipped 2 previous similar messages [10920.928451] LustreError: 408797:0:(osp_dev.c:692:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [10926.129539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10930.751258] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [10934.651799] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10946.020242] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10946.027967] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10946.041514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10948.729127] LustreError: 410362:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10948.736563] LustreError: 410362:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [10948.831845] Lustre: server umount lustre-OST0000 complete [10957.867603] Lustre: server umount lustre-MDT0000 complete [10961.042213] LustreError: 407125:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351158 with bad export cookie 11832063040862343591 [10961.046137] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10961.049726] LustreError: 407125:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10961.288759] Lustre: server umount lustre-MDT0001 complete [10969.470167] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [10972.206154] Key type lgssc unregistered [10972.565523] LNet: 411303:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10972.586051] LNetError: 411303:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10973.607136] LNet: Removed LNI 192.168.202.127@tcp [10974.476390] Key type .llcrypt unregistered [10974.477883] Key type ._llcrypt unregistered [10991.940805] Key type ._llcrypt registered [10991.942255] Key type .llcrypt registered [10992.020721] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [11005.129627] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [11006.267525] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11006.306771] alg: No test for adler32 (adler32-zlib) [11007.380247] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [11007.597543] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [11009.279524] Key type lgssc registered [11010.428776] Lustre: Echo OBD driver; http://www.lustre.org/ [11017.907952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11023.075684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11028.593374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11035.047024] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11041.524806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11041.572722] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11041.581980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11042.780458] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11042.838589] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11042.900607] Lustre: lustre-MDT0000: new disk, initializing [11042.988554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11043.001198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11046.086620] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11055.402676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11055.472363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11055.551298] Lustre: 414426:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11055.595642] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11055.601995] Lustre: Skipped 1 previous similar message [11055.653324] Lustre: lustre-MDT0001: new disk, initializing [11055.739895] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11055.816846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11055.820471] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11059.063345] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11065.805352] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11068.926164] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11075.224418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11075.295703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11075.495139] Lustre: lustre-OST0000: new disk, initializing [11075.498265] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11075.563632] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11080.526611] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11083.898745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11083.903581] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11084.002477] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11087.874324] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11091.866550] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11092.058902] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11095.210679] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11095.472630] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11099.617142] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11099.620866] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11099.630448] LustreError: Skipped 1 previous similar message [11099.637361] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11103.301979] LustreError: 417364:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [11103.516789] Lustre: server umount lustre-OST0000 complete [11109.858048] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11109.864702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11109.873058] Lustre: Skipped 2 previous similar messages [11109.880922] Lustre: Skipped 1 previous similar message [11113.409570] LustreError: 417565:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [11113.416049] LustreError: 417565:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11115.488976] LustreError: 417565:0:(obd_class.h:479:obd_check_dev()) Device 8 not setup [11115.491757] LustreError: 417565:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11115.540485] Lustre: server umount lustre-MDT0000 complete [11118.568908] LustreError: 414418:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351315 with bad export cookie 13000180152279243886 [11118.576856] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11118.593293] LustreError: 414418:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [11118.730764] LustreError: 417768:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11118.733883] LustreError: 417768:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [11118.964426] Lustre: server umount lustre-MDT0001 complete [11126.881210] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11128.059612] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 22:48:44 (1763351324) [11134.130658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11134.488499] LustreError: 418490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11134.500721] LustreError: 418490:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11134.563158] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11138.211223] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11139.557870] LustreError: 418490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11144.673485] LustreError: 418490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11145.863320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11146.351405] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11150.488487] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11155.430981] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11159.154723] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11164.758923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11165.068707] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11169.357785] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11173.418314] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11174.809474] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11181.482605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11186.656915] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11186.662537] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11186.671834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11189.730816] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11189.733964] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11189.739415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11190.851659] LustreError: 420933:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11190.861336] LustreError: 420933:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [11191.075539] Lustre: server umount lustre-OST0000 complete [11197.410384] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11197.424752] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11197.445721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11197.449707] Lustre: Skipped 1 previous similar message [11201.306568] LustreError: 421135:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [11201.314481] LustreError: 421135:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11201.512912] Lustre: server umount lustre-MDT0000 complete [11205.088354] LustreError: 419494:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11205.118159] LustreError: 419494:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11205.594491] LustreError: 418470:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351402 with bad export cookie 13000180152279244768 [11205.600419] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11205.605460] LustreError: 418470:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [11206.065198] Lustre: server umount lustre-MDT0001 complete [11216.268590] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [11219.541517] Key type lgssc unregistered [11219.884986] LNet: 421869:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11219.901393] LNetError: 421869:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11219.919321] LNet: Removed LNI 192.168.202.127@tcp [11220.686789] Key type .llcrypt unregistered [11220.688270] Key type ._llcrypt unregistered [11244.766619] Key type ._llcrypt registered [11244.768243] Key type .llcrypt registered [11244.875391] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [11245.186322] Key type .llcrypt unregistered [11245.187936] Key type ._llcrypt unregistered [11260.656075] Key type ._llcrypt registered [11260.657734] Key type .llcrypt registered [11260.764219] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [11261.806213] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11261.833875] alg: No test for adler32 (adler32-zlib) [11262.983366] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [11263.261719] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [11264.964279] Key type lgssc registered [11266.289326] Lustre: Echo OBD driver; http://www.lustre.org/ [11273.451768] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11273.468215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11274.802977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11278.888574] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11283.939406] LustreError: 424037:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11283.959399] LustreError: 424037:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11287.874753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11288.142407] LustreError: 424038:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11288.166649] LustreError: 424038:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11288.195325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11293.226889] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11293.664856] LustreError: 424038:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11293.695314] LustreError: 424038:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11298.472740] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11298.788898] LustreError: 424033:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11298.795981] LustreError: 424033:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11298.815279] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11303.914860] LustreError: 424037:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11303.958912] LustreError: 424037:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11305.505438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11306.046282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11310.077874] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11319.265337] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11319.268951] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11319.277716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11321.313610] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11321.316424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11321.322801] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11323.587678] LustreError: 426093:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [11323.753168] Lustre: server umount lustre-OST0000 complete [11331.554579] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11331.558075] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11331.564862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11331.567726] Lustre: Skipped 2 previous similar messages [11333.540254] LustreError: 426294:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [11333.543558] LustreError: 426294:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11333.611822] Lustre: server umount lustre-MDT0000 complete [11336.649663] LustreError: 424018:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351533 with bad export cookie 11553289091355448924 [11336.657021] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11336.765985] LustreError: 426495:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11336.776762] LustreError: 426495:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11336.888989] Lustre: server umount lustre-MDT0001 complete [11345.038507] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [11347.613847] Key type lgssc unregistered [11347.861799] LNet: 427026:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11347.874615] LNetError: 427026:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11347.889273] LNet: Removed LNI 192.168.202.127@tcp [11348.584287] Key type .llcrypt unregistered [11348.588145] Key type ._llcrypt unregistered [11368.483935] Key type ._llcrypt registered [11368.485525] Key type .llcrypt registered [11368.601602] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [11369.350074] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11369.390050] alg: No test for adler32 (adler32-zlib) [11370.400338] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [11370.588359] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [11372.232395] Key type lgssc registered [11373.249942] Lustre: Echo OBD driver; http://www.lustre.org/ [11378.967632] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11378.990589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11394.527588] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11399.647761] LustreError: 428397:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.127@tcp: failed processing log, type 4: rc = -110 [11426.335317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11430.917592] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11435.889753] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11440.995476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11441.432799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11441.480988] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11444.146348] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11447.779576] LustreError: 429320:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11447.788051] LustreError: 429320:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [11450.259765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11450.478099] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11453.749770] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11458.638150] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11461.816090] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11467.234234] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11467.238290] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11467.244953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11470.817239] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11470.833411] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11470.853074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11472.655124] LustreError: 430678:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [11472.811923] Lustre: server umount lustre-OST0000 complete [11476.451214] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11476.455062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11476.464845] Lustre: Skipped 1 previous similar message [11476.468844] Lustre: Skipped 2 previous similar messages [11481.571604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11481.578451] Lustre: Skipped 1 previous similar message [11482.315591] LustreError: 430879:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [11482.321168] LustreError: 430879:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11482.441436] Lustre: server umount lustre-MDT0000 complete [11485.969613] LustreError: 428404:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351683 with bad export cookie 14257949966547927253 [11485.996571] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11486.193601] LustreError: 431080:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11486.209691] LustreError: 431080:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11486.445811] Lustre: server umount lustre-MDT0001 complete [11496.331177] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [11498.709327] Key type lgssc unregistered [11498.959922] LNet: 431611:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11498.968247] LNetError: 431611:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11498.983821] LNet: Removed LNI 192.168.202.127@tcp [11499.671140] Key type .llcrypt unregistered [11499.673978] Key type ._llcrypt unregistered [11520.196624] Key type ._llcrypt registered [11520.197971] Key type .llcrypt registered [11520.289341] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [11521.217892] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11521.285852] alg: No test for adler32 (adler32-zlib) [11522.413285] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [11522.647143] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [11524.319191] Key type lgssc registered [11525.483373] Lustre: Echo OBD driver; http://www.lustre.org/ [11532.575705] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11532.600595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11534.046972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11538.000397] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11544.555838] LustreError: 432991:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11544.577758] LustreError: 432991:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11546.341710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11547.015525] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11551.108905] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11556.185946] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11559.461419] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11565.794526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11566.080377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11570.730420] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11575.629257] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11579.392699] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11582.944714] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11582.948220] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11582.955774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11584.487119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11584.488722] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11584.514903] Lustre: Skipped 1 previous similar message [11586.242225] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.27@tcp (stopping) [11586.246177] Lustre: Skipped 2 previous similar messages [11588.966323] LustreError: 435239:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [11589.027462] Lustre: server umount lustre-MDT0000 complete [11589.609591] LustreError: 432986:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11589.621887] LustreError: 432986:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11591.192316] LustreError: 432972:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.27@tcp arrived at 1763351788 with bad export cookie 18382537639885117218 [11592.950285] LustreError: 432972:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763351790 with bad export cookie 18382537639885116553 [11592.959649] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11592.974981] LustreError: 432972:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [11594.720588] LustreError: 432990:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11594.723456] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11594.729880] LustreError: 432990:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11594.748386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11597.730607] LustreError: 432990:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11597.750125] LustreError: 432990:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11599.045057] LustreError: 435441:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11599.048171] LustreError: 435441:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11599.284277] Lustre: server umount lustre-OST0000 complete [11602.855417] LustreError: 432990:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11602.863037] LustreError: 432990:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11605.433803] LustreError: 435796:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [11605.438158] LustreError: 435796:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11605.565362] Lustre: server umount lustre-MDT0001 complete [11614.723136] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [11617.458966] Key type lgssc unregistered [11617.730163] LNet: 436328:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11617.743960] LNetError: 436328:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11617.769097] LNet: Removed LNI 192.168.202.127@tcp [11618.514863] Key type .llcrypt unregistered [11618.517131] Key type ._llcrypt unregistered [11635.985139] Key type ._llcrypt registered [11635.986965] Key type .llcrypt registered [11636.070322] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [11636.815238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11636.836451] alg: No test for adler32 (adler32-zlib) [11637.826538] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [11637.984134] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [11639.639217] Key type lgssc registered [11640.489288] Lustre: Echo OBD driver; http://www.lustre.org/ [11646.925163] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11646.943669] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11648.301969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11651.191973] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11657.413760] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11657.814085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11661.138550] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11665.109915] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11668.362859] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11674.079782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11674.159590] LustreError: 439303:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11674.183375] LustreError: 439303:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11674.356446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11678.826352] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11679.761019] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11682.872701] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11687.913163] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11687.925288] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11689.667452] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.27@tcp (stopping) [11689.672272] Lustre: Skipped 1 previous similar message [11691.883232] LustreError: 439949:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [11691.940140] Lustre: server umount lustre-MDT0001 complete [11693.034875] LustreError: 437703:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11693.053806] LustreError: 437703:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11698.145475] LustreError: 437703:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11698.145956] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11698.179122] Lustre: Skipped 1 previous similar message [11698.184335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11702.464240] LustreError: 440151:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11702.467663] LustreError: 440151:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11702.561646] Lustre: server umount lustre-OST0000 complete [11703.265762] LustreError: 437703:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11705.594718] LustreError: 440352:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [11705.607532] LustreError: 440352:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11705.784563] Lustre: server umount lustre-MDT0000 complete [11717.770799] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [11719.994231] Key type lgssc unregistered [11720.261293] LNet: 441036:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11720.269866] LNetError: 441036:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11720.288370] LNet: Removed LNI 192.168.202.127@tcp [11721.046910] Key type .llcrypt unregistered [11721.051877] Key type ._llcrypt unregistered [11742.031498] Key type ._llcrypt registered [11742.033242] Key type .llcrypt registered [11742.135389] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [11752.081456] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [11752.780229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11752.804988] alg: No test for adler32 (adler32-zlib) [11753.727438] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [11753.886655] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [11755.511498] Key type lgssc registered [11756.237459] Lustre: Echo OBD driver; http://www.lustre.org/ [11761.844822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11766.303746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11770.772131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11775.278202] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11781.193337] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [11787.071347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11792.153771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11792.185563] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11792.200260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11793.385941] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11793.413700] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11793.465914] Lustre: lustre-MDT0000: new disk, initializing [11793.541790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11793.555155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11796.520689] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11802.413903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11807.702955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11807.752349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11807.912327] Lustre: lustre-OST0000: new disk, initializing [11807.916078] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11807.919074] Lustre: Skipped 1 previous similar message [11807.975921] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11809.440191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [11809.452057] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [11809.494699] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11812.186286] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11817.768480] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11824.578147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11829.538896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11829.603505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11829.676474] Lustre: 444988:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11829.747203] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11829.808581] Lustre: lustre-MDT0001: new disk, initializing [11829.931446] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11829.974375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [11829.985912] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [11833.039479] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11839.724609] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [11839.857484] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [11842.455890] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [11842.599629] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [11847.649339] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11847.651277] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11847.657679] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11847.664896] Lustre: Skipped 1 previous similar message [11847.671212] Lustre: Skipped 1 previous similar message [11849.410296] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.27@tcp (stopping) [11851.667554] LustreError: 448406:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [11851.743602] Lustre: server umount lustre-MDT0001 complete [11857.898557] LustreError: 444994:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11857.909563] LustreError: 444994:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [11863.010070] LustreError: 448409:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11863.019590] LustreError: 448409:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11868.131118] LustreError: 444998:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11868.137120] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11868.144916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11872.195168] LustreError: 448690:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [11872.203490] LustreError: 448690:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11872.275888] Lustre: server umount lustre-OST0000 complete [11873.252251] LustreError: 444998:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11880.731978] LustreError: 448892:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [11880.735238] LustreError: 448892:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11880.847265] Lustre: server umount lustre-MDT0000 complete [11889.323411] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [11890.798895] Key type lgssc unregistered [11890.969485] LNet: 449577:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11890.973611] LNetError: 449577:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11890.986496] LNet: Removed LNI 192.168.202.127@tcp [11891.409316] Key type .llcrypt unregistered [11891.411083] Key type ._llcrypt unregistered [11912.244744] Key type ._llcrypt registered [11912.246294] Key type .llcrypt registered [11912.326370] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [11912.536293] Key type .llcrypt unregistered [11912.538133] Key type ._llcrypt unregistered [11922.891240] Key type ._llcrypt registered [11922.893218] Key type .llcrypt registered [11922.982658] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [11923.612245] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11923.664809] alg: No test for adler32 (adler32-zlib) [11924.625336] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [11924.752229] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [11926.375193] Key type lgssc registered [11927.114747] Lustre: Echo OBD driver; http://www.lustre.org/ [11931.553209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11935.396526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11938.915696] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11943.038903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11943.071458] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11943.082957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11944.235498] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11944.256209] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11944.306346] Lustre: lustre-MDT0000: new disk, initializing [11944.368440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11944.384405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11946.561862] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11952.977632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11953.022550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11953.089372] Lustre: 452897:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11953.157757] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11953.162685] Lustre: Skipped 1 previous similar message [11953.212422] Lustre: lustre-MDT0001: new disk, initializing [11953.286609] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11953.317217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11953.322030] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11955.717886] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11960.361853] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11962.325496] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11965.912126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11965.952436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11966.081567] Lustre: lustre-OST0000: new disk, initializing [11966.084743] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11966.133540] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11967.188987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11967.194391] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11967.260089] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11969.489885] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [11974.773211] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11979.235730] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11979.240681] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11979.255511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11982.821763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11982.821853] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11982.826540] Lustre: Skipped 1 previous similar message [11982.836629] Lustre: Skipped 1 previous similar message [11984.601762] LustreError: 455339:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [11984.683135] Lustre: server umount lustre-MDT0000 complete [11986.754427] LustreError: 453679:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352183 with bad export cookie 16211570921417360699 [11986.757660] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11986.762152] LustreError: 453679:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [11986.844680] LustreError: 455541:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11986.846742] LustreError: 455541:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11986.951325] Lustre: server umount lustre-MDT0001 complete [11999.096196] LustreError: 455742:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [11999.099963] LustreError: 455742:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11999.185279] Lustre: server umount lustre-OST0000 complete [12006.256944] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 23:03:22 (1763352202) [12006.899985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12011.507818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12011.785163] LustreError: 456572:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12011.801103] LustreError: 456572:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12011.843112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12014.500271] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12017.126556] LustreError: 456572:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12020.212884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12020.431233] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12022.522587] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12025.250215] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12027.083226] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12031.040689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12031.092406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12031.216712] Lustre: Found index 0 for lustre-OST0000, updating log [12031.337626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12034.798359] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12037.486677] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12037.797148] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12042.721866] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12042.731513] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12042.740994] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12044.835192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12044.838952] Lustre: Skipped 1 previous similar message [12046.133390] LustreError: 458827:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12046.135566] LustreError: 458827:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [12046.217058] Lustre: server umount lustre-OST0000 complete [12047.630676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12050.914205] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12050.919740] Lustre: Skipped 1 previous similar message [12050.919758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12050.928107] Lustre: Skipped 3 previous similar messages [12054.740076] LustreError: 459082:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [12054.743571] LustreError: 459082:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12054.792947] Lustre: server umount lustre-MDT0000 complete [12056.651677] LustreError: 456553:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352253 with bad export cookie 16211570921417362092 [12056.654978] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12056.656634] LustreError: 456553:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [12056.807233] Lustre: server umount lustre-MDT0001 complete [12061.463527] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 23:04:17 (1763352257) [12062.042429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12062.569179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12063.118258] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12063.697262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12064.222113] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12064.757580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12065.243921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12065.847358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12066.423259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12066.944021] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12067.468076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12067.957492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12068.489546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12069.069133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12069.605375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12070.165434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12070.662692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12071.189589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12071.726273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12072.340663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12072.895763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12073.404679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12073.985934] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12074.475753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12074.962340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12075.437880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12075.936979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12076.457031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12076.928280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12077.444521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12077.890474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12078.344650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12078.808083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12079.291300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12079.869821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12080.391120] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12080.887460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12081.409939] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12081.902415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12082.407510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12082.951740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12083.455680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12083.928466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12084.371240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12084.813680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12085.263208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12085.703756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12086.219157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12086.679135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12087.131993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12087.611338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12091.264899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12091.488454] LustreError: 462672:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12091.525271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12093.309301] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12096.993602] LustreError: 462672:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12097.476941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12098.850224] LNet: 1 local NIs in recovery (showing 1): 192.168.202.127@tcp [12099.649281] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12102.450522] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12104.151563] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12107.401485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12107.446056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12107.553074] Lustre: Found index 0 for lustre-OST0000, updating log [12110.348085] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12112.811875] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12113.889686] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12113.889944] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12113.899417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12113.902035] Lustre: Skipped 2 previous similar messages [12115.778938] LustreError: 464940:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12115.782463] LustreError: 464940:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [12115.832753] Lustre: server umount lustre-OST0000 complete [12123.616699] LustreError: 462671:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12123.633631] LustreError: 462671:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12123.652088] Lustre: server umount lustre-MDT0000 complete [12125.578097] LustreError: 462651:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352322 with bad export cookie 16211570921417363149 [12125.579491] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12125.587757] LustreError: 462651:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [12125.787894] Lustre: server umount lustre-MDT0001 complete [12127.455098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12135.598551] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [12140.078402] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [12144.441652] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12147.790556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12151.074444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12154.124180] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12157.119535] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 23:05:53 (1763352353) [12159.263248] LNet: 1 local NIs in recovery (showing 1): 192.168.202.127@tcp [12173.580522] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [12175.021782] Key type lgssc unregistered [12175.174639] LNet: 468926:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12175.179910] LNetError: 468926:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12302.181924] LNet: Removed LNI 192.168.202.127@tcp [12302.594931] Key type .llcrypt unregistered [12302.596810] Key type ._llcrypt unregistered [12310.553261] Key type ._llcrypt registered [12310.554593] Key type .llcrypt registered [12310.606053] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [12310.962980] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12310.989974] alg: No test for adler32 (adler32-zlib) [12311.867356] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [12311.992456] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [12313.591171] Key type lgssc registered [12314.149039] Lustre: Echo OBD driver; http://www.lustre.org/ [12316.381358] LNet: Added LNI 192.168.202.127@tcp100 [8/256/0/180] [12316.812115] LNet: Added LNI 192.168.202.127@tcp101 [8/256/0/180] [12317.234620] LNet: Added LNI 192.168.202.127@tcp102 [8/256/0/180] [12317.648243] LNet: Added LNI 192.168.202.127@tcp103 [8/256/0/180] [12318.090511] LNet: Added LNI 192.168.202.127@tcp104 [8/256/0/180] [12318.526609] LNet: Added LNI 192.168.202.127@tcp105 [8/256/0/180] [12319.034368] LNet: Added LNI 192.168.202.127@tcp106 [8/256/0/180] [12319.521520] LNet: Added LNI 192.168.202.127@tcp107 [8/256/0/180] [12319.988565] LNet: Added LNI 192.168.202.127@tcp108 [8/256/0/180] [12320.497874] LNet: Added LNI 192.168.202.127@tcp109 [8/256/0/180] [12320.989779] LNet: Added LNI 192.168.202.127@tcp110 [8/256/0/180] [12321.443648] LNet: Added LNI 192.168.202.127@tcp111 [8/256/0/180] [12321.905715] LNet: Added LNI 192.168.202.127@tcp112 [8/256/0/180] [12322.298483] LNet: Added LNI 192.168.202.127@tcp113 [8/256/0/180] [12322.767733] LNet: Added LNI 192.168.202.127@tcp114 [8/256/0/180] [12323.259194] LNet: Added LNI 192.168.202.127@tcp115 [8/256/0/180] [12323.714570] LNet: Added LNI 192.168.202.127@tcp116 [8/256/0/180] [12324.162807] LNet: Added LNI 192.168.202.127@tcp117 [8/256/0/180] [12324.656851] LNet: Added LNI 192.168.202.127@tcp118 [8/256/0/180] [12325.098296] LNet: Added LNI 192.168.202.127@tcp119 [8/256/0/180] [12325.583351] LNet: Added LNI 192.168.202.127@tcp120 [8/256/0/180] [12326.043550] LNet: Added LNI 192.168.202.127@tcp121 [8/256/0/180] [12326.489496] LNet: Added LNI 192.168.202.127@tcp122 [8/256/0/180] [12326.908948] LNet: Added LNI 192.168.202.127@tcp123 [8/256/0/180] [12327.375991] LNet: Added LNI 192.168.202.127@tcp124 [8/256/0/180] [12327.887670] LNet: Added LNI 192.168.202.127@tcp125 [8/256/0/180] [12328.405872] LNet: Added LNI 192.168.202.127@tcp126 [8/256/0/180] [12328.845561] LNet: Added LNI 192.168.202.127@tcp127 [8/256/0/180] [12329.309623] LNet: Added LNI 192.168.202.127@tcp128 [8/256/0/180] [12329.759573] LNet: Added LNI 192.168.202.127@tcp129 [8/256/0/180] [12330.176156] LNet: Added LNI 192.168.202.127@tcp130 [8/256/0/180] [12330.606082] LNet: Added LNI 192.168.202.127@tcp131 [8/256/0/180] [12331.012231] LNet: Added LNI 192.168.202.127@tcp132 [8/256/0/180] [12331.418941] LNet: Added LNI 192.168.202.127@tcp133 [8/256/0/180] [12331.854458] LNet: Added LNI 192.168.202.127@tcp134 [8/256/0/180] [12332.319133] LNet: Added LNI 192.168.202.127@tcp135 [8/256/0/180] [12335.248212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12335.278247] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12335.284935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12336.334251] Lustre: 471917:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.127@tcp: can't add NID 192.168.202.127@tcp130: rc = -75 [12336.372599] LNetError: 471940:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.127 [12336.379502] LNetError: Protocol error connecting to 192.168.202.127@tcp120 at host 192.168.202.127:988: is it running a compatible version of Lustre? [12336.448716] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12336.472384] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12336.514219] Lustre: lustre-MDT0000: new disk, initializing [12336.564628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12336.576739] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12336.607067] LNet: 2 local NIs in recovery (showing 2): 192.168.202.127@tcp120, 192.168.202.127@tcp123 [12337.440719] LNetError: 469378:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.127 [12337.445735] LNetError: 469378:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 4 previous similar messages [12337.449287] LNetError: Protocol error connecting to 192.168.202.127@tcp104 at host 192.168.202.127:988: is it running a compatible version of Lustre? [12337.455548] LNetError: Skipped 4 previous similar messages [12338.355423] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12339.687353] LNetError: 469376:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.127 [12339.692129] LNetError: 469376:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 2 previous similar messages [12339.696256] LNetError: Protocol error connecting to 192.168.202.127@tcp120 at host 192.168.202.127:988: is it running a compatible version of Lustre? [12339.701106] LNetError: Skipped 2 previous similar messages [12343.777314] LNetError: 471938:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.127 [12343.782620] LNetError: Protocol error connecting to 192.168.202.127@tcp120 at host 192.168.202.127:988: is it running a compatible version of Lustre? [12343.835458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12343.868884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12343.921458] Lustre: 471936:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12343.958780] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12343.961898] Lustre: Skipped 1 previous similar message [12344.003340] Lustre: lustre-MDT0001: new disk, initializing [12344.018300] Lustre: 472721:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.127@tcp: can't add NID 192.168.202.127@tcp130: rc = -75 [12344.022253] Lustre: 472721:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [12344.075822] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12344.094455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12344.098477] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12345.758497] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12349.452834] Lustre: 473199:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.127@tcp: can't add NID 192.168.202.127@tcp130: rc = -75 [12349.457123] Lustre: 473199:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [12349.517451] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12350.997274] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12351.970530] LNetError: 471946:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.127 [12351.977248] LNetError: Protocol error connecting to 192.168.202.127@tcp120 at host 192.168.202.127:988: is it running a compatible version of Lustre? [12353.972219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12354.010314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12354.137755] Lustre: lustre-OST0000: new disk, initializing [12354.140955] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12354.185975] Lustre: 473692:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.127@tcp: can't add NID 192.168.202.127@tcp130: rc = -75 [12354.189970] Lustre: 473692:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [12354.213433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12356.601213] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12358.644537] Lustre: 474160:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.127@tcp: can't add NID 192.168.202.127@tcp130: rc = -75 [12358.648499] Lustre: 474160:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [12358.677743] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12358.682450] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12358.739783] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12360.229376] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12363.746131] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12363.746464] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12363.749715] LustreError: Skipped 1 previous similar message [12363.756081] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12363.760663] Lustre: Skipped 1 previous similar message [12365.234140] LustreError: 474527:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [12365.294613] Lustre: server umount lustre-OST0000 complete [12366.240314] LNetError: 469377:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.127 [12366.245551] LNetError: Protocol error connecting to 192.168.202.127@tcp119 at host 192.168.202.127:988: is it running a compatible version of Lustre? [12368.868723] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12368.869844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12368.876358] Lustre: Skipped 2 previous similar messages [12368.881964] Lustre: Skipped 1 previous similar message [12373.214947] LustreError: 474728:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [12373.253677] LustreError: 474728:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12373.297383] Lustre: server umount lustre-MDT0000 complete [12375.173148] LustreError: 473686:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352572 with bad export cookie 1342882845385024345 [12375.176290] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12375.177182] LustreError: 473686:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [12375.253713] LustreError: 474930:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12375.256280] LustreError: 474930:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [12375.322770] Lustre: server umount lustre-MDT0001 complete [12379.825996] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [12381.055599] Key type lgssc unregistered [12381.236696] LNet: 475469:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12381.242078] LNetError: 475469:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12381.244579] LNet: Removed LNI 192.168.202.127@tcp [12382.303208] LNet: Removed LNI 192.168.202.127@tcp100 [12382.305525] LNetError: 475469:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12382.309461] LNetError: 475469:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [12382.314073] LNet: Removed LNI 192.168.202.127@tcp101 [12383.327442] LNet: Removed LNI 192.168.202.127@tcp102 [12383.329367] LNetError: 475469:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12383.332285] LNetError: 475469:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [12383.335845] LNet: Removed LNI 192.168.202.127@tcp103 [12384.351188] LNet: Removed LNI 192.168.202.127@tcp104 [12385.375188] LNet: Removed LNI 192.168.202.127@tcp105 [12385.377236] LNetError: 475469:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12385.381470] LNetError: 475469:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [12385.386462] LNet: Removed LNI 192.168.202.127@tcp106 [12386.399212] LNet: Removed LNI 192.168.202.127@tcp107 [12387.423478] LNet: Removed LNI 192.168.202.127@tcp108 [12388.447233] LNet: Removed LNI 192.168.202.127@tcp109 [12389.473040] LNet: Removed LNI 192.168.202.127@tcp110 [12389.475147] LNetError: 475469:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12389.479209] LNetError: 475469:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [12389.484921] LNet: Removed LNI 192.168.202.127@tcp111 [12390.495663] LNet: Removed LNI 192.168.202.127@tcp112 [12391.519656] LNet: Removed LNI 192.168.202.127@tcp113 [12392.543262] LNet: Removed LNI 192.168.202.127@tcp114 [12393.567162] LNet: Removed LNI 192.168.202.127@tcp115 [12394.591177] LNet: Removed LNI 192.168.202.127@tcp116 [12395.615163] LNet: Removed LNI 192.168.202.127@tcp117 [12396.639171] LNet: Removed LNI 192.168.202.127@tcp118 [12397.663189] LNet: Removed LNI 192.168.202.127@tcp119 [12397.665292] LNetError: 475469:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12397.668269] LNetError: 475469:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [12397.671345] LNet: Removed LNI 192.168.202.127@tcp120 [12398.690128] LNet: Removed LNI 192.168.202.127@tcp121 [12399.711122] LNet: Removed LNI 192.168.202.127@tcp122 [12400.735248] LNet: Removed LNI 192.168.202.127@tcp123 [12400.737089] LNet: Removed LNI 192.168.202.127@tcp124 [12400.738822] LNet: Removed LNI 192.168.202.127@tcp125 [12400.740764] LNet: Removed LNI 192.168.202.127@tcp126 [12400.742520] LNet: Removed LNI 192.168.202.127@tcp127 [12400.744256] LNet: Removed LNI 192.168.202.127@tcp128 [12400.745937] LNet: Removed LNI 192.168.202.127@tcp129 [12400.749160] LNet: Removed LNI 192.168.202.127@tcp130 [12400.751857] LNet: Removed LNI 192.168.202.127@tcp131 [12400.753924] LNet: Removed LNI 192.168.202.127@tcp132 [12400.755498] LNet: Removed LNI 192.168.202.127@tcp133 [12400.757292] LNet: Removed LNI 192.168.202.127@tcp134 [12400.765633] LNet: Removed LNI 192.168.202.127@tcp135 [12401.095825] Key type .llcrypt unregistered [12401.097600] Key type ._llcrypt unregistered [12411.272478] Key type ._llcrypt registered [12411.273758] Key type .llcrypt registered [12411.322391] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [12418.814342] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [12419.233250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12419.242590] alg: No test for adler32 (adler32-zlib) [12420.141253] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [12420.256660] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [12421.863240] Key type lgssc registered [12422.404364] Lustre: Echo OBD driver; http://www.lustre.org/ [12426.764071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12429.590054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12432.401612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12435.223833] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12438.193775] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 23:10:34 (1763352634) [12446.042890] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [12447.571242] Key type lgssc unregistered [12447.769295] LNet: 479484:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12447.772692] LNetError: 479484:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12447.784544] LNet: Removed LNI 192.168.202.127@tcp [12448.224312] Key type .llcrypt unregistered [12448.226037] Key type ._llcrypt unregistered [12456.797693] Key type ._llcrypt registered [12456.799176] Key type .llcrypt registered [12456.849996] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [12457.350632] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12457.358353] alg: No test for adler32 (adler32-zlib) [12458.245943] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [12458.361420] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [12459.967125] Key type lgssc registered [12460.457820] Lustre: Echo OBD driver; http://www.lustre.org/ [12465.161541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12465.190134] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12465.197690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12466.331027] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12466.354641] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12466.398803] Lustre: lustre-MDT0000: new disk, initializing [12466.453468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12466.467330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12468.517470] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12474.276027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12474.309529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12474.354588] Lustre: 480654:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12474.388945] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12474.391056] Lustre: Skipped 1 previous similar message [12474.435460] Lustre: lustre-MDT0001: new disk, initializing [12474.482408] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12474.501451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12474.507101] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12476.358617] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12480.084392] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12481.781820] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12485.022877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12485.050485] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12485.162844] Lustre: lustre-OST0000: new disk, initializing [12485.165959] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12485.203028] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12487.539655] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12491.127363] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12492.304648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12492.309662] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12492.369980] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12492.667449] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing lnet_if_list [12494.803860] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [12494.819130] LNet: Added LNI 192.168.202.127@tcp10 [8/256/0/180] [12497.296229] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [12497.312410] LNet: Added LNI 192.168.202.127@tcp11 [8/256/0/180] [12499.274564] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [12499.290609] LNet: Added LNI 192.168.202.127@tcp12 [8/256/0/180] [12501.264977] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [12501.283113] LNet: Added LNI 192.168.202.127@tcp13 [8/256/0/180] [12503.145700] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [12503.161537] LNet: Added LNI 192.168.202.127@tcp14 [8/256/0/180] [12505.064352] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [12505.081569] LNet: Added LNI 192.168.202.127@tcp15 [8/256/0/180] [12507.172224] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [12507.189767] LNet: Added LNI 192.168.202.127@tcp16 [8/256/0/180] [12509.336309] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [12509.354647] LNet: Added LNI 192.168.202.127@tcp17 [8/256/0/180] [12511.529705] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [12511.545995] LNet: Added LNI 192.168.202.127@tcp18 [8/256/0/180] [12513.637224] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [12513.652118] LNet: Added LNI 192.168.202.127@tcp19 [8/256/0/180] [12515.749218] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [12515.766338] LNet: Added LNI 192.168.202.127@tcp20 [8/256/0/180] [12517.978238] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [12517.995525] LNet: Added LNI 192.168.202.127@tcp21 [8/256/0/180] [12520.209865] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [12520.225519] LNet: Added LNI 192.168.202.127@tcp22 [8/256/0/180] [12522.649664] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [12522.675815] LNet: Added LNI 192.168.202.127@tcp23 [8/256/0/180] [12525.132612] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [12525.157837] LNet: Added LNI 192.168.202.127@tcp24 [8/256/0/180] [12527.526671] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [12527.545676] LNet: Added LNI 192.168.202.127@tcp25 [8/256/0/180] [12530.115311] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [12530.130785] LNet: Added LNI 192.168.202.127@tcp26 [8/256/0/180] [12532.816957] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [12532.832513] LNet: Added LNI 192.168.202.127@tcp27 [8/256/0/180] [12532.840843] LNetError: 479914:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.127 [12532.847587] LNetError: Protocol error connecting to 192.168.202.127@tcp26 at host 192.168.202.127:988: is it running a compatible version of Lustre? [12535.716791] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [12535.733945] LNet: Added LNI 192.168.202.127@tcp28 [8/256/0/180] [12535.738284] LNetError: 479917:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.127 [12535.743888] LNetError: Protocol error connecting to 192.168.202.127@tcp26 at host 192.168.202.127:988: is it running a compatible version of Lustre? [12538.386814] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [12538.401907] LNet: Added LNI 192.168.202.127@tcp29 [8/256/0/180] [12538.407526] LNetError: 479916:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.127 [12538.413729] LNetError: Protocol error connecting to 192.168.202.127@tcp28 at host 192.168.202.127:988: is it running a compatible version of Lustre? [12539.295296] LNet: 1 local NIs in recovery (showing 1): 192.168.202.127@tcp28 [12540.989720] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [12541.005537] LNet: Added LNI 192.168.202.127@tcp30 [8/256/0/180] [12541.345355] LNetError: 479915:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.127 [12541.350302] LNetError: 479915:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 1 previous similar message [12541.354247] LNetError: Protocol error connecting to 192.168.202.127@tcp28 at host 192.168.202.127:988: is it running a compatible version of Lustre? [12541.359929] LNetError: Skipped 1 previous similar message [12543.569712] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [12543.583519] LNet: Added LNI 192.168.202.127@tcp31 [8/256/0/180] [12545.442497] LNetError: 479914:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.127 [12545.447746] LNetError: 479914:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 2 previous similar messages [12545.451456] LNetError: Protocol error connecting to 192.168.202.127@tcp28 at host 192.168.202.127:988: is it running a compatible version of Lustre? [12545.455983] LNetError: Skipped 2 previous similar messages [12546.209546] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [12546.225877] LNet: Added LNI 192.168.202.127@tcp32 [8/256/0/180] [12548.910834] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [12548.927247] LNet: Added LNI 192.168.202.127@tcp33 [8/256/0/180] [12551.830025] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [12551.846606] LNet: Added LNI 192.168.202.127@tcp34 [8/256/0/180] [12553.633921] LNetError: 479914:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.127 [12553.638612] LNetError: 479914:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 10 previous similar messages [12553.643672] LNetError: Protocol error connecting to 192.168.202.127@tcp30 at host 192.168.202.127:988: is it running a compatible version of Lustre? [12553.650637] LNetError: Skipped 10 previous similar messages [12554.941344] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [12554.955983] LNet: Added LNI 192.168.202.127@tcp35 [8/256/0/180] [12557.912787] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [12557.929266] LNet: Added LNI 192.168.202.127@tcp36 [8/256/0/180] [12560.546703] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [12560.560826] LNet: Added LNI 192.168.202.127@tcp37 [8/256/0/180] [12563.264572] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [12563.280424] LNet: Added LNI 192.168.202.127@tcp38 [8/256/0/180] [12566.051548] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [12566.065773] LNet: Added LNI 192.168.202.127@tcp39 [8/256/0/180] [12568.901089] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [12568.916318] LNet: Added LNI 192.168.202.127@tcp40 [8/256/0/180] [12570.018956] LNetError: 494122:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.127 [12570.024149] LNetError: 494122:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 16 previous similar messages [12570.027713] LNetError: Protocol error connecting to 192.168.202.127@tcp37 at host 192.168.202.127:988: is it running a compatible version of Lustre? [12570.033219] LNetError: Skipped 16 previous similar messages [12572.165204] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [12572.182544] LNet: Added LNI 192.168.202.127@tcp41 [8/256/0/180] [12575.439645] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [12575.458129] LNet: Added LNI 192.168.202.127@tcp42 [8/256/0/180] [12578.852942] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [12578.868626] LNet: Added LNI 192.168.202.127@tcp43 [8/256/0/180] [12582.459286] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [12582.473964] LNet: Added LNI 192.168.202.127@tcp44 [8/256/0/180] [12585.443323] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [12585.457111] LNet: Added LNI 192.168.202.127@tcp45 [8/256/0/180] [12588.597560] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [12588.611249] LNet: Added LNI 192.168.202.127@tcp46 [8/256/0/180] [12592.128316] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [12592.145798] LNet: Added LNI 192.168.202.127@tcp47 [8/256/0/180] [12595.510292] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [12595.526813] LNet: Added LNI 192.168.202.127@tcp48 [8/256/0/180] [12598.963406] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [12598.979383] LNet: Added LNI 192.168.202.127@tcp49 [8/256/0/180] [12599.712391] LNet: 8 local NIs in recovery (showing 8): 192.168.202.127@tcp46, 192.168.202.127@tcp44, 192.168.202.127@tcp41, 192.168.202.127@tcp35, 192.168.202.127@tcp40, 192.168.202.127@tcp31, 192.168.202.127@tcp30, 192.168.202.127@tcp29 [12602.716925] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [12602.733174] LNet: Added LNI 192.168.202.127@tcp50 [8/256/0/180] [12602.738757] LNetError: 479917:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.127 [12602.743412] LNetError: 479917:0:(socklnd_cb.c:1806:ksocknal_recv_hello()) Skipped 32 previous similar messages [12602.747875] LNetError: Protocol error connecting to 192.168.202.127@tcp47 at host 192.168.202.127:988: is it running a compatible version of Lustre? [12602.752908] LNetError: Skipped 32 previous similar messages [12615.138034] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12615.142481] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12615.150125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12615.153240] Lustre: Skipped 1 previous similar message [12617.906467] LustreError: 504854:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [12617.961409] Lustre: server umount lustre-OST0000 complete [12620.258288] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12620.259219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12620.263694] Lustre: Skipped 2 previous similar messages [12620.268685] Lustre: Skipped 1 previous similar message [12625.376415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12625.378968] Lustre: Skipped 1 previous similar message [12630.496486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12630.499402] Lustre: Skipped 1 previous similar message [12634.079216] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12634.123561] LustreError: 505057:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [12634.126604] LustreError: 505057:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12634.158889] Lustre: server umount lustre-MDT0000 complete [12635.724839] LustreError: 480646:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352832 with bad export cookie 1640555106619860238 [12635.725951] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12635.733280] LustreError: 480646:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [12635.810819] LustreError: 505262:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12635.812932] LustreError: 505262:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [12635.892392] Lustre: server umount lustre-MDT0001 complete [12640.317376] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [12641.671224] Key type lgssc unregistered [12641.868705] LNet: 505801:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12641.873762] LNetError: 505801:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12641.877113] LNet: Removed LNI 192.168.202.127@tcp [12642.911182] LNet: Removed LNI 192.168.202.127@tcp10 [12642.913327] LNetError: 505801:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12642.916681] LNetError: 505801:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [12642.920741] LNet: Removed LNI 192.168.202.127@tcp11 [12643.935433] LNet: Removed LNI 192.168.202.127@tcp12 [12643.937670] LNetError: 505801:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12643.940982] LNetError: 505801:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [12643.944537] LNet: Removed LNI 192.168.202.127@tcp13 [12644.959238] LNet: Removed LNI 192.168.202.127@tcp14 [12645.983187] LNet: Removed LNI 192.168.202.127@tcp15 [12645.985272] LNetError: 505801:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12645.989333] LNetError: 505801:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [12645.992333] LNet: Removed LNI 192.168.202.127@tcp16 [12647.007234] LNet: Removed LNI 192.168.202.127@tcp17 [12648.031223] LNet: Removed LNI 192.168.202.127@tcp18 [12649.055255] LNet: Removed LNI 192.168.202.127@tcp19 [12650.079413] LNet: Removed LNI 192.168.202.127@tcp20 [12650.081213] LNetError: 505801:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12650.085026] LNetError: 505801:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [12650.088542] LNet: Removed LNI 192.168.202.127@tcp21 [12651.103210] LNet: Removed LNI 192.168.202.127@tcp22 [12652.127431] LNet: Removed LNI 192.168.202.127@tcp23 [12653.151632] LNet: Removed LNI 192.168.202.127@tcp24 [12654.175435] LNet: Removed LNI 192.168.202.127@tcp25 [12655.199619] LNet: Removed LNI 192.168.202.127@tcp26 [12656.223387] LNet: Removed LNI 192.168.202.127@tcp27 [12657.247207] LNet: Removed LNI 192.168.202.127@tcp28 [12657.249144] LNet: Removed LNI 192.168.202.127@tcp29 [12657.251075] LNet: Removed LNI 192.168.202.127@tcp30 [12657.252689] LNet: Removed LNI 192.168.202.127@tcp31 [12658.271269] LNet: Removed LNI 192.168.202.127@tcp32 [12658.273429] LNetError: 505801:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12658.276817] LNetError: 505801:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Skipped 11 previous similar messages [12658.279827] LNet: Removed LNI 192.168.202.127@tcp33 [12659.295321] LNet: Removed LNI 192.168.202.127@tcp34 [12659.297539] LNet: Removed LNI 192.168.202.127@tcp35 [12660.319227] LNet: Removed LNI 192.168.202.127@tcp36 [12661.343477] LNet: Removed LNI 192.168.202.127@tcp37 [12662.367417] LNet: Removed LNI 192.168.202.127@tcp38 [12663.391696] LNet: Removed LNI 192.168.202.127@tcp39 [12664.415235] LNet: Removed LNI 192.168.202.127@tcp40 [12664.417349] LNet: Removed LNI 192.168.202.127@tcp41 [12665.439352] LNet: Removed LNI 192.168.202.127@tcp42 [12666.463198] LNet: Removed LNI 192.168.202.127@tcp43 [12667.487660] LNet: Removed LNI 192.168.202.127@tcp44 [12668.511605] LNet: Removed LNI 192.168.202.127@tcp45 [12668.514143] LNet: Removed LNI 192.168.202.127@tcp46 [12668.516884] LNet: Removed LNI 192.168.202.127@tcp47 [12669.535160] LNet: Removed LNI 192.168.202.127@tcp48 [12669.537068] LNet: Removed LNI 192.168.202.127@tcp49 [12669.545611] LNet: Removed LNI 192.168.202.127@tcp50 [12669.852098] Key type .llcrypt unregistered [12669.853482] Key type ._llcrypt unregistered [12679.053569] Key type ._llcrypt registered [12679.055315] Key type .llcrypt registered [12679.100370] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [12679.500518] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12679.611991] alg: No test for adler32 (adler32-zlib) [12680.508429] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [12680.616913] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [12682.215157] Key type lgssc registered [12682.674023] Lustre: Echo OBD driver; http://www.lustre.org/ [12685.982125] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12685.995277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12687.242452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12688.866448] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12692.449933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12692.602736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12694.253034] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12696.535873] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12698.024963] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12700.858620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12701.005050] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12703.285507] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12705.469224] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12720.958067] loop0: detected capacity change from 0 to 204800000 [12721.083419] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12721.088517] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12721.094431] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12721.120051] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12724.003750] loop0: detected capacity change from 0 to 204800000 [12724.075393] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12724.081489] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12724.089867] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12724.155399] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12726.503661] loop0: detected capacity change from 0 to 204800000 [12729.000949] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12729.033973] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12729.097179] Lustre: fs15246-OST0000: new disk, initializing [12729.099784] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [12729.134333] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12731.801956] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12736.671203] loop1: detected capacity change from 0 to 204800000 [12739.219807] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12739.248251] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12739.292745] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [12739.320478] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [12739.365622] Lustre: fs15246-MDT0000: new disk, initializing [12739.437193] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12739.448924] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12739.494336] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [12740.512026] Lustre: 511423:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000aed515b5 x1849009506386944/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1763352953 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [12741.339877] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12749.608515] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12749.613056] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12749.629887] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12751.331813] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12751.339445] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [12756.454283] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [12757.437254] LustreError: 512549:0:(obd_class.h:479:obd_check_dev()) Device 30 not setup [12757.458672] Lustre: server umount fs15246-MDT0000 complete [12761.160092] LustreError: 513050:0:(obd_class.h:479:obd_check_dev()) Device 32 not setup [12761.163158] LustreError: 513050:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12761.182625] Lustre: server umount fs15246-OST0000 complete [12766.696426] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12766.707128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12766.710595] Lustre: Skipped 1 previous similar message [12771.503159] LustreError: 513551:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12771.505376] LustreError: 513551:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [12771.557621] Lustre: server umount lustre-OST0000 complete [12774.879933] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12774.882961] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12774.887784] Lustre: Skipped 1 previous similar message [12774.890174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12776.929137] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12779.336288] LustreError: 513753:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [12779.338342] LustreError: 513753:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12779.366516] Lustre: server umount lustre-MDT0000 complete [12780.919684] LustreError: 507004:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763352978 with bad export cookie 201571075365708734 [12780.921661] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12780.929021] LustreError: 507004:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [12781.111271] Lustre: server umount lustre-MDT0001 complete [12785.431814] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [12786.794172] Key type lgssc unregistered [12786.975066] LNet: 514486:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12786.980179] LNetError: 514486:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12786.987620] LNet: Removed LNI 192.168.202.127@tcp [12787.386885] Key type .llcrypt unregistered [12787.388751] Key type ._llcrypt unregistered [12796.243771] Key type ._llcrypt registered [12796.245309] Key type .llcrypt registered [12796.291488] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [12796.630739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12796.638197] alg: No test for adler32 (adler32-zlib) [12797.516127] Lustre: Lustre: Build Version: 2.16.61_49_g8566d6d [12797.617426] LNet: Added LNI 192.168.202.127@tcp [8/256/0/180] [12799.207218] Key type lgssc registered [12799.680676] Lustre: Echo OBD driver; http://www.lustre.org/ [12802.916242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12804.961819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12807.131809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12809.170613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12815.996070] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [12819.994365] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [12823.908988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12826.758820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12829.502979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12832.201884] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12834.958402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12834.980922] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12834.987376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12836.085053] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12836.100912] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12836.142992] Lustre: lustre-MDT0000: new disk, initializing [12836.186823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12836.198445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12837.783387] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12842.677617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12842.708100] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12842.757215] Lustre: 518889:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12842.786844] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12842.789923] Lustre: Skipped 1 previous similar message [12842.829621] Lustre: lustre-MDT0001: new disk, initializing [12842.871602] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12842.895085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12842.899848] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12844.498664] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12848.167549] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12849.639510] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12852.491147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12852.526829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12852.658481] Lustre: lustre-OST0000: new disk, initializing [12852.662132] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12852.710156] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12855.157166] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_default_debug -1 all [12858.755594] Lustre: DEBUG MARKER: oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12860.363844] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12861.967873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12861.972283] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12862.014526] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12862.496506] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [12864.124073] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12864.216154] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12867.041854] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12867.044093] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12867.045160] LustreError: Skipped 1 previous similar message [12867.050978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12867.055765] Lustre: Skipped 1 previous similar message [12871.101635] LustreError: 521839:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [12871.166809] Lustre: server umount lustre-OST0000 complete [12873.697143] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12873.702415] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12873.707881] Lustre: Skipped 1 previous similar message [12873.714359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12877.280204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12877.281419] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12878.986350] LustreError: 522040:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [12878.989470] LustreError: 522040:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12879.031597] Lustre: server umount lustre-MDT0000 complete [12880.855988] LustreError: 518881:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1763353078 with bad export cookie 7961596111209958007 [12880.859533] LustreError: MGC192.168.202.127@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12880.860505] LustreError: 518881:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [12880.936518] LustreError: 522242:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12880.939664] LustreError: 522242:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [12881.027517] Lustre: server umount lustre-MDT0001 complete [12890.406333] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing set_hostid [12894.278559] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing load_modules_local [12898.535902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12901.087717] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12903.696289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12906.395671] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12907.110706] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12667 sec ========= 23:18:23 (1763353103) [12907.805447] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 23:18:24 (1763353104) === [12909.310968] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 23:18:25 (1763353105) === [12926.193720] Lustre: DEBUG MARKER: oleg227-server.virtnet: executing unload_modules_local [12927.432448] Key type lgssc unregistered [12927.572471] LNet: 527091:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12927.575368] LNetError: 527091:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12927.583390] LNet: Removed LNI 192.168.202.127@tcp [12927.937813] Key type .llcrypt unregistered [12927.939560] Key type ._llcrypt unregistered