[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-8.fc42 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 688638344 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE2439 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE22D5 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 002295 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE2349 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE23D9 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2411 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe22d5-0xbffe2348] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe22d4] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe2349-0xbffe23d8] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe23d9-0xbffe2410] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2411-0xbffe2438] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.002314] x2apic enabled [ 0.003005] Switched APIC routing to physical x2apic. [ 0.004008] kvm-guest: setup PV IPIs [ 0.007440] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009008] pid_max: default: 32768 minimum: 301 [ 0.010111] LSM: Security Framework initializing [ 0.011034] Yama: becoming mindful. [ 0.012031] SELinux: Initializing. [ 0.013049] *** VALIDATE selinux *** [ 0.022132] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027440] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029101] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030084] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031087] *** VALIDATE tmpfs *** [ 0.033380] *** VALIDATE proc *** [ 0.034178] *** VALIDATE cgroup *** [ 0.035005] *** VALIDATE cgroup2 *** [ 0.037149] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039111] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041024] Spectre V2 : User space: Vulnerable [ 0.042006] Speculative Store Bypass: Vulnerable [ 0.045062] debug: unmapping init [mem 0xffffffffba259000-0xffffffffba260fff] [ 0.047262] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048726] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049018] ... version: 2 [ 0.050082] ... bit width: 48 [ 0.051007] ... generic registers: 4 [ 0.052007] ... value mask: 0000ffffffffffff [ 0.053008] ... max period: 00007fffffffffff [ 0.054009] ... fixed-purpose events: 3 [ 0.055008] ... event mask: 000000070000000f [ 0.056253] rcu: Hierarchical SRCU implementation. [ 0.058549] smp: Bringing up secondary CPUs ... [ 0.059484] x86: Booting SMP configuration: [ 0.060015] .... node #0, CPUs: #1 #2 #3 [ 0.065306] smp: Brought up 1 node, 4 CPUs [ 0.067021] smpboot: Max logical packages: 1 [ 0.068010] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.102060] node 0 deferred pages initialised in 32ms [ 0.106078] devtmpfs: initialized [ 0.107233] x86/mm: Memory block size: 128MB [ 0.108000] gcov: version magic: 0x41383552 [ 0.111393] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.115070] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.118380] pinctrl core: initialized pinctrl subsystem [ 0.120266] [ 0.120719] ************************************************************* [ 0.123012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.126010] ** ** [ 0.129013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.133014] ** ** [ 0.136011] ** This means that this kernel is built to expose internal ** [ 0.140018] ** IOMMU data structures, which may compromise security on ** [ 0.144011] ** your system. ** [ 0.147043] ** ** [ 0.150015] ** If you see this message and you are not debugging the ** [ 0.153012] ** kernel, report this immediately to your vendor! ** [ 0.156051] ** ** [ 0.158010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.159041] ************************************************************* [ 0.163180] NET: Registered protocol family 16 [ 0.164641] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.168057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.171051] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.174085] cpuidle: using governor menu [ 0.175868] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.179007] PCI: Using configuration type 1 for base access [ 0.181126] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.189045] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.190050] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.192023] cryptd: max_cpu_qlen set to 1000 [ 0.194452] ACPI: Added _OSI(Module Device) [ 0.195000] ACPI: Added _OSI(Processor Device) [ 0.195000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.195000] ACPI: Added _OSI(Processor Aggregator Device) [ 0.199000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.208542] ACPI: Interpreter enabled [ 0.209050] ACPI: PM: (supports S0 S3 S4 S5) [ 0.211009] ACPI: Using IOAPIC for interrupt routing [ 0.212141] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.216531] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.228396] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.231035] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.233015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.236076] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.242831] acpiphp: Slot [2] registered [ 0.243122] acpiphp: Slot [5] registered [ 0.245132] acpiphp: Slot [6] registered [ 0.247529] acpiphp: Slot [7] registered [ 0.249231] acpiphp: Slot [8] registered [ 0.251559] acpiphp: Slot [9] registered [ 0.253260] acpiphp: Slot [10] registered [ 0.255105] acpiphp: Slot [3] registered [ 0.257119] acpiphp: Slot [4] registered [ 0.258084] acpiphp: Slot [11] registered [ 0.260067] acpiphp: Slot [12] registered [ 0.261058] acpiphp: Slot [13] registered [ 0.262084] acpiphp: Slot [14] registered [ 0.263152] acpiphp: Slot [15] registered [ 0.265118] acpiphp: Slot [16] registered [ 0.266079] acpiphp: Slot [17] registered [ 0.268089] acpiphp: Slot [18] registered [ 0.269000] acpiphp: Slot [19] registered [ 0.269000] acpiphp: Slot [20] registered [ 0.269000] acpiphp: Slot [21] registered [ 0.269076] acpiphp: Slot [22] registered [ 0.270000] acpiphp: Slot [23] registered [ 0.272095] acpiphp: Slot [24] registered [ 0.273085] acpiphp: Slot [25] registered [ 0.275075] acpiphp: Slot [26] registered [ 0.276093] acpiphp: Slot [27] registered [ 0.278123] acpiphp: Slot [28] registered [ 0.280089] acpiphp: Slot [29] registered [ 0.281088] acpiphp: Slot [30] registered [ 0.282061] acpiphp: Slot [31] registered [ 0.284055] PCI host bridge to bus 0000:00 [ 0.285015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.287016] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.289016] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.291013] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.293040] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.295022] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.297204] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.300000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.305000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.325047] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.328535] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.330012] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.331000] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.332008] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.334138] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.337984] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.341045] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.344976] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.351016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.364023] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.370025] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.378589] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.384018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.393026] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.417025] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.430616] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.442020] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.453022] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.481023] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.492865] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.505022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.517024] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.536021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.546351] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.561024] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.566019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.594021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.603345] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.610019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.623021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.638024] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.685000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.697081] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.709021] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.727023] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.740950] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.745954] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.749437] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.750436] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.751200] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.753135] iommu: Default domain type: Passthrough [ 0.754000] SCSI subsystem initialized [ 0.754000] ACPI: bus type USB registered [ 0.758148] usbcore: registered new interface driver usbfs [ 0.761083] usbcore: registered new interface driver hub [ 0.765114] usbcore: registered new device driver usb [ 0.768190] pps_core: LinuxPPS API ver. 1 registered [ 0.771012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.774388] PTP clock support registered [ 0.777342] EDAC MC: Ver: 3.0.0 [ 0.778019] PCI: Using ACPI for IRQ routing [ 0.780129] NetLabel: Initializing [ 0.783014] NetLabel: domain hash size = 128 [ 0.786011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.789473] NetLabel: unlabeled traffic allowed by default [ 0.793008] vgaarb: loaded [ 0.794606] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.796007] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.803326] clocksource: Switched to clocksource kvm-clock [ 0.920584] VFS: Disk quotas dquot_6.6.0 [ 0.922294] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.924377] *** VALIDATE ramfs *** [ 0.925572] *** VALIDATE hugetlbfs *** [ 0.927284] pnp: PnP ACPI init [ 0.930657] pnp: PnP ACPI: found 6 devices [ 0.947237] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.950380] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.952572] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.954845] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.957096] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.959324] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.961401] NET: Registered protocol family 2 [ 0.963607] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.968514] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.971872] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.977281] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.980960] TCP: Hash tables configured (established 65536 bind 65536) [ 0.983301] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.986078] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.988328] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.991154] NET: Registered protocol family 1 [ 0.994561] RPC: Registered named UNIX socket transport module. [ 0.996664] RPC: Registered udp transport module. [ 0.998487] RPC: Registered tcp transport module. [ 0.999746] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.002328] NET: Registered protocol family 44 [ 1.004327] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.006824] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.008571] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.010922] PCI: CLS 0 bytes, default 64 [ 1.012528] Unpacking initramfs... [ 3.088798] debug: unmapping init [mem 0xffff9421fcc54000-0xffff9421fffbffff] [ 3.092473] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.094785] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.097535] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.641475] Initialise system trusted keyrings [ 3.642965] Key type blacklist registered [ 3.645034] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.654523] zbud: loaded [ 3.657344] *** VALIDATE nfs *** [ 3.658345] *** VALIDATE nfs4 *** [ 3.659788] pstore: using deflate compression [ 3.663192] Platform Keyring initialized [ 3.762410] NET: Registered protocol family 38 [ 3.763801] Key type asymmetric registered [ 3.764921] Asymmetric key parser 'x509' registered [ 3.766472] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.768829] io scheduler mq-deadline registered [ 3.769967] io scheduler kyber registered [ 3.771426] io scheduler bfq registered [ 3.772847] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.775291] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.777470] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.780196] ACPI: Power Button [PWRF] [ 3.784790] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.792863] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.805223] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.812578] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.827196] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.855174] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.882828] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.887664] Non-volatile memory driver v1.3 [ 3.889264] Linux agpgart interface v0.103 [ 3.917800] virtio_blk virtio1: [vda] 134032 512-byte logical blocks (68.6 MB/65.4 MiB) [ 3.920058] vda: detected capacity change from 0 to 68624384 [ 3.932862] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.935448] vdb: detected capacity change from 0 to 1073741824 [ 3.956943] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.959592] vdc: detected capacity change from 0 to 2621440000 [ 3.997364] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.999900] vdd: detected capacity change from 0 to 2621440000 [ 4.018604] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.021224] vde: detected capacity change from 0 to 4294967296 [ 4.051499] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.054546] vdf: detected capacity change from 0 to 4294967296 [ 4.060255] libphy: Fixed MDIO Bus: probed [ 4.069334] usbcore: registered new interface driver usbserial_generic [ 4.072057] usbserial: USB Serial support registered for generic [ 4.074592] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.080434] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.082570] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.085555] mousedev: PS/2 mouse device common for all mice [ 4.088779] rtc_cmos 00:05: RTC can wake from S4 [ 4.090645] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.091577] rtc_cmos 00:05: registered as rtc0 [ 4.099327] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.101337] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.104282] intel_pstate: CPU model not supported [ 4.107329] hid: raw HID events driver (C) Jiri Kosina [ 4.114227] usbcore: registered new interface driver usbhid [ 4.118481] usbhid: USB HID core driver [ 4.120175] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.120859] drop_monitor: Initializing network drop monitor service [ 4.129332] Initializing XFRM netlink socket [ 4.131412] NET: Registered protocol family 10 [ 4.134570] Segment Routing with IPv6 [ 4.136164] NET: Registered protocol family 17 [ 4.137870] mpls_gso: MPLS GSO support [ 4.143740] RAS: Correctable Errors collector initialized. [ 4.145469] AVX version of gcm_enc/dec engaged. [ 4.146755] AES CTR mode by8 optimization enabled [ 4.208721] sched_clock: Marking stable (4208589451, 0)->(5350281655, -1141692204) [ 4.214367] registered taskstats version 1 [ 4.216220] Loading compiled-in X.509 certificates [ 4.217796] zswap: loaded using pool lzo/zbud [ 4.238762] Key type big_key registered [ 4.262806] Key type encrypted registered [ 4.264146] ima: No TPM chip found, activating TPM-bypass! [ 4.268136] ima: Allocated hash algorithm: sha1 [ 4.270921] ima: No architecture policies found [ 4.275241] evm: Initialising EVM extended attributes: [ 4.279224] evm: security.selinux [ 4.280254] evm: security.ima [ 4.282662] evm: security.capability [ 4.284670] evm: HMAC attrs: 0x1 [ 4.291853] rtc_cmos 00:05: setting system clock to 2026-01-02 00:46:00 UTC (1767314760) [ 4.306858] debug: unmapping init [mem 0xffffffffbb203000-0xffffffffbb3fffff] [ 4.314712] debug: unmapping init [mem 0xffffffffb9f82000-0xffffffffba258fff] [ 4.322079] Write protecting the kernel read-only data: 28672k [ 4.326217] debug: unmapping init [mem 0xffffffffb8603000-0xffffffffb87fffff] [ 4.330874] debug: unmapping init [mem 0xffffffffb8f14000-0xffffffffb8ffffff] [ 4.405793] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.419620] systemd[1]: Detected virtualization kvm. [ 4.421623] systemd[1]: Detected architecture x86-64. [ 4.426156] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.471832] systemd[1]: No hostname configured. [ 4.475745] systemd[1]: Set hostname to . [ 4.479563] random: systemd: uninitialized urandom read (16 bytes read) [ 4.484320] systemd[1]: Initializing machine ID from random generator. [ 4.605947] random: ln: uninitialized urandom read (6 bytes read) [ 4.778739] random: systemd: uninitialized urandom read (16 bytes read) [ 4.785035] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.808451] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.835378] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Reached target Local File Systems. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.439231] device-mapper: uevent: version 1.0.3 [ 6.444150] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 8.104066] virtio_net virtio0 ens2: renamed from eth0 [ 8.147103] random: fast init done [ 8.276618] scsi host0: ata_piix [ 8.350804] scsi host1: ata_piix [ 8.355675] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.360509] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 12.795115] random: crng init done [ 12.796570] random: 7 urandom warning(s) missed due to ratelimiting [ 12.840123] dracut-initqueue[583]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 14.347802] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 16.185808] printk: systemd: 26 output lines suppressed due to ratelimiting [ 17.181886] SELinux: Disabled at runtime. [ 17.336660] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 17.362100] systemd[1]: Detected virtualization kvm. [ 17.366125] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 18.800646] systemd[1]: initrd-switch-root.service: Succeeded. [ 18.807818] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 18.826584] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 18.833794] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 18.841826] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 18.865408] systemd[1]: Starting Journal Service... Starting Journal Service... [ 18.885957] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Stopped target Switch Root. [ 19.103423] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. Mounting Kernel Debug File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 19.788009] hrtimer: interrupt took 5586025 ns [ 19.834332] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 20.532961] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 20.565110] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 20.815617] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 20.935463] EDAC sbridge: Ver: 1.1.2 [ 23.575663] Key type dns_resolver registered [ 24.126904] NFS: Registering the id_resolver key type [ 24.129210] Key type id_resolver registered [ 24.130439] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (6s / no limit) [** ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg405-server login: [ 67.790094] libcfs: loading out-of-tree module taints kernel. [ 67.826846] Key type ._llcrypt registered [ 67.828873] Key type .llcrypt registered [ 67.913731] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 132.436784] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 133.390483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 133.404426] alg: No test for adler32 (adler32-zlib) [ 134.612607] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [ 135.142348] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 136.815251] Key type lgssc registered [ 137.974839] Lustre: Echo OBD driver; http://www.lustre.org/ [ 163.630618] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 164.334082] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 164.512437] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 165.774563] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 165.844787] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 251.357136] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 274.792185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 274.813460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 274.888756] Lustre: lustre-MDT0001: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 274.903337] Lustre: Skipped 1 previous similar message [ 274.905978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 274.934792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 278.152964] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 278.217143] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 278.303592] Lustre: lustre-MDT0001: new disk, initializing [ 278.415656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 278.474937] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 278.482793] Lustre: 15498:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 278.491158] LustreError: 15498:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 278.554648] Lustre: 15480:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 278.948915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 278.971874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 280.040268] Lustre: 15542:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@0000000054b01652 x1853163979152768/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1767315052 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 283.204360] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 283.731997] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 285.103524] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 285.109539] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 292.452660] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 293.040923] Lustre: 16688:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 318.879395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 318.928292] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 318.983225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 319.012289] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 319.193452] Lustre: lustre-OST0000: new disk, initializing [ 319.196273] Lustre: Skipped 1 previous similar message [ 319.198977] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 319.210827] Lustre: Skipped 2 previous similar messages [ 319.265202] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 326.550436] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 326.799414] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 327.389157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 327.406348] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 327.480594] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 327.503981] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000400 [ 341.520543] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 350.630952] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 368.921725] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing check_logdir /tmp/testlogs/ [ 374.761451] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing yml_node [ 385.722377] Lustre: DEBUG MARKER: Client: 2.17.0.RC4 [ 388.843277] Lustre: DEBUG MARKER: MDS: 2.17.0.RC4 [ 392.651558] Lustre: DEBUG MARKER: OSS: 2.17.0.RC4 [ 394.754586] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Jan 1 19:52:28 EST 2026 [ 421.306369] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 423.176457] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 440.290931] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 440.315410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 440.323199] Lustre: Skipped 3 previous similar messages [ 445.414148] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 445.420757] Lustre: Skipped 3 previous similar messages [ 446.121305] LustreError: 22213:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 446.275875] Lustre: server umount lustre-MDT0001 complete [ 446.548719] Lustre: server umount lustre-MDT0000 complete [ 456.618531] LustreError: 15454:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767315212 with bad export cookie 17580868116482118843 [ 456.623166] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 456.631873] LustreError: 15454:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 456.673949] LustreError: 23194:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 456.679899] LustreError: 23194:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 456.775618] Lustre: server umount lustre-OST0001 complete [ 457.070896] Lustre: server umount lustre-OST0000 complete [ 477.156650] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 555.424777] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 579.512438] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 580.052487] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 581.739547] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 585.478664] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 610.559147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 610.670867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 611.203485] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 611.304869] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 611.311926] Lustre: Skipped 1 previous similar message [ 611.480507] Lustre: lustre-MDT0000: new disk, initializing [ 611.484478] Lustre: Skipped 1 previous similar message [ 611.669355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 611.672431] Lustre: Skipped 1 previous similar message [ 611.699382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 611.703248] Lustre: Skipped 1 previous similar message [ 616.692810] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 641.074950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 641.157511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 641.253967] Lustre: 31122:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 641.334386] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 641.341943] Lustre: Skipped 1 previous similar message [ 641.418925] Lustre: lustre-MDT0001: new disk, initializing [ 641.541292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 641.584398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 641.605894] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 641.615755] Lustre: Skipped 1 previous similar message [ 646.735599] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 656.742379] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 661.045988] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 682.815105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 682.870209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 683.250593] Lustre: lustre-OST0000: new disk, initializing [ 683.256610] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 683.413985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 684.425253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 684.442269] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 684.612755] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 691.098898] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 700.926497] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 706.705343] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 706.914130] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 711.618876] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 711.939920] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 715.237559] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 715.239790] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 715.240817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 715.264917] Lustre: Skipped 4 previous similar messages [ 720.191131] LustreError: 35097:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 720.199259] LustreError: 35097:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 720.406099] Lustre: server umount lustre-OST0000 complete [ 728.547702] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 728.561951] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 728.586891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 728.597329] Lustre: Skipped 1 previous similar message [ 730.605962] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 735.713321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 735.730063] Lustre: Skipped 3 previous similar messages [ 740.319155] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 740.465971] LustreError: 35347:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 740.474738] LustreError: 35347:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 740.853822] Lustre: server umount lustre-MDT0000 complete [ 746.186780] LustreError: 31113:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767315502 with bad export cookie 17580868116482120236 [ 746.190576] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 746.202797] LustreError: 31113:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 746.360815] LustreError: 35598:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 746.381076] LustreError: 35598:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 746.787463] Lustre: server umount lustre-MDT0001 complete [ 751.392287] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 19:58:25 (1767315505) [ 754.842461] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 757.248765] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 19:58:30 (1767315510) [ 768.411101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 768.899571] LustreError: 36510:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 768.923904] LustreError: 36510:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 769.011593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 773.564958] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 774.113541] LustreError: 36510:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 779.237145] LustreError: 36510:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 784.359913] LustreError: 36510:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 786.284393] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 792.212637] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 798.881228] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 804.941454] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 824.762360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 826.086272] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 826.098383] Lustre: Skipped 1 previous similar message [ 834.657199] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 841.534992] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 849.888432] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 849.897968] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 849.908770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 849.918800] Lustre: Skipped 3 previous similar messages [ 855.887834] LustreError: 39142:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 855.895405] LustreError: 39142:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 856.234672] Lustre: server umount lustre-OST0000 complete [ 863.716768] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 863.721721] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 863.730090] Lustre: Skipped 1 previous similar message [ 869.662703] Lustre: server umount lustre-MDT0000 complete [ 870.374883] LustreError: 36505:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 870.397311] LustreError: 36505:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 876.415392] LustreError: 36490:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767315632 with bad export cookie 17580868116482121125 [ 876.427141] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 876.442672] LustreError: 36490:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 876.752520] LustreError: 39644:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 876.764886] LustreError: 39644:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 877.335757] Lustre: server umount lustre-MDT0001 complete [ 896.953516] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 900.086752] Key type lgssc unregistered [ 900.406737] LNet: 40322:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 900.417480] LNetError: 40322:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 901.478680] LNet: Removed LNI 192.168.204.105@tcp [ 902.886130] Key type .llcrypt unregistered [ 902.887579] Key type ._llcrypt unregistered [ 1004.694681] Key type ._llcrypt registered [ 1004.696645] Key type .llcrypt registered [ 1004.807544] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 1005.868778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1005.887274] alg: No test for adler32 (adler32-zlib) [ 1007.260286] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [ 1007.675775] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 1009.415294] Key type lgssc registered [ 1011.350866] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1027.398784] loop: module loaded [ 1027.518206] loop0: detected capacity change from 0 to 204800000 [ 1027.916682] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 1027.932116] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 1028.567224] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 1028.696573] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1037.197391] loop0: detected capacity change from 0 to 204800000 [ 1037.503377] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 1037.515278] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 1037.535273] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 1037.764271] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1052.730872] loop0: detected capacity change from 0 to 204800000 [ 1059.796524] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1059.870986] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1059.895326] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1061.194375] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1061.315236] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1061.450577] Lustre: lustre-MDT0000: new disk, initializing [ 1061.584322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1061.638499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1067.632888] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1084.544649] loop1: detected capacity change from 0 to 204800000 [ 1091.089283] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1091.156632] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1091.444316] Lustre: lustre-OST07c6: new disk, initializing [ 1091.451699] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 1091.466542] Lustre: Skipped 1 previous similar message [ 1091.538307] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 1096.894096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 1096.906779] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 1096.967449] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 1099.474591] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1106.013462] Lustre: 46045:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 1106.020726] Lustre: 46045:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 1106.023194] Lustre: 46045:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 1106.034801] Lustre: 46045:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 1106.042873] Lustre: 46045:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 1106.050181] Lustre: 46045:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 1122.349209] Lustre: 46046:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 1122.361483] Lustre: 46046:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 1122.376938] Lustre: 46046:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1122.384047] Lustre: 46046:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 1122.396252] Lustre: 46046:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 1122.410146] Lustre: 46046:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1127.757608] Lustre: 46046:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 1127.766374] Lustre: 46046:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 1 previous similar message [ 1127.772328] Lustre: 46046:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 1127.782390] Lustre: 46046:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1127.793312] Lustre: 46046:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1127.799821] Lustre: 46046:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1127.805806] Lustre: 46046:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 1127.811647] Lustre: 46046:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1127.818981] Lustre: 46046:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 1127.826505] Lustre: 46046:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1127.833777] Lustre: 46046:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1127.840783] Lustre: 46046:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1132.966581] Lustre: 47342:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 1132.986270] Lustre: 47342:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 1 previous similar message [ 1132.992692] Lustre: 47342:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 1133.000566] Lustre: 47342:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1133.007583] Lustre: 47342:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1133.015505] Lustre: 47342:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1133.022521] Lustre: 47342:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/4 [ 1133.031238] Lustre: 47342:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1133.042801] Lustre: 47342:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 1133.050295] Lustre: 47342:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1133.066164] Lustre: 47342:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1133.079584] Lustre: 47342:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1137.835763] Lustre: 47342:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 1137.847132] Lustre: 47342:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 1 previous similar message [ 1137.853742] Lustre: 47342:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 1137.860737] Lustre: 47342:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1137.865695] Lustre: 47342:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1137.870961] Lustre: 47342:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1137.876138] Lustre: 47342:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/0, punch: 0/0/0, quota 4/150/0 [ 1137.881416] Lustre: 47342:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1137.885254] Lustre: 47342:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 1137.889674] Lustre: 47342:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1137.894760] Lustre: 47342:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1137.910909] Lustre: 47342:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 1 previous similar message [ 1149.033100] Lustre: 47342:0:(osd_internal.h:1459:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 1149.046216] Lustre: 47342:0:(osd_internal.h:1459:osd_trans_exec_op()) Skipped 5 previous similar messages [ 1149.054612] Lustre: 47342:0:(osd_handler.c:2096:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 1149.077712] Lustre: 47342:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 1149.084968] Lustre: 47342:0:(osd_handler.c:2103:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 1149.093347] Lustre: 47342:0:(osd_handler.c:2103:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 1149.103319] Lustre: 47342:0:(osd_handler.c:2113:osd_trans_dump_creds()) write: 2/23/1, punch: 0/0/0, quota 4/150/0 [ 1149.109747] Lustre: 47342:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 1149.120958] Lustre: 47342:0:(osd_handler.c:2120:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 1149.126756] Lustre: 47342:0:(osd_handler.c:2120:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 1149.133068] Lustre: 47342:0:(osd_handler.c:2127:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 1149.140226] Lustre: 47342:0:(osd_handler.c:2127:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 1199.591675] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1199.597421] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1199.622537] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 1204.713158] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 1209.828892] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 1212.383188] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1212.412592] LustreError: 48522:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 1212.511351] Lustre: server umount lustre-OST07c6 complete [ 1223.436872] LustreError: 49024:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 1223.449222] LustreError: 49024:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 1223.763419] Lustre: server umount lustre-MDT0000 complete [ 1248.238649] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 1250.914956] Key type lgssc unregistered [ 1251.174499] LNet: 50003:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1251.190312] LNetError: 50003:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1252.215702] LNet: Removed LNI 192.168.204.105@tcp [ 1252.867498] Key type .llcrypt unregistered [ 1252.869393] Key type ._llcrypt unregistered [ 1331.855828] Key type ._llcrypt registered [ 1331.857914] Key type .llcrypt registered [ 1331.965947] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 1332.734342] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1332.753598] alg: No test for adler32 (adler32-zlib) [ 1333.896248] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [ 1334.107243] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 1335.777908] Key type lgssc registered [ 1337.057801] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1344.800514] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1344.816412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1346.445671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1350.667389] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1356.773447] LustreError: 54317:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1356.796606] LustreError: 54317:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1361.828648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1361.895293] LustreError: 54317:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1362.209906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1366.273484] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1373.402495] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1378.030289] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1390.175296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1390.445259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1391.479960] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1397.311085] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1404.210919] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1422.307364] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1422.319833] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1422.339254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1422.340811] Lustre: Skipped 1 previous similar message [ 1423.724421] LustreError: 57153:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1424.110745] Lustre: server umount lustre-OST0000 complete [ 1432.547226] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1432.561797] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1432.564811] Lustre: Skipped 1 previous similar message [ 1435.865323] LustreError: 57404:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1435.868919] LustreError: 57404:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1435.927737] Lustre: server umount lustre-MDT0000 complete [ 1437.668241] LustreError: 54317:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1437.693235] LustreError: 54317:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1441.720757] LustreError: 54298:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767316197 with bad export cookie 2976321543661189631 [ 1441.727259] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1441.735857] LustreError: 54298:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1441.956845] LustreError: 57654:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1441.962469] LustreError: 57654:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1442.233947] Lustre: server umount lustre-MDT0001 complete [ 1457.524818] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 1460.628823] Key type lgssc unregistered [ 1460.958788] LNet: 58330:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1460.967721] LNetError: 58330:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1461.005604] LNet: Removed LNI 192.168.204.105@tcp [ 1461.968361] Key type .llcrypt unregistered [ 1461.970919] Key type ._llcrypt unregistered [ 1540.166465] Key type ._llcrypt registered [ 1540.170733] Key type .llcrypt registered [ 1540.321752] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 1541.874910] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1541.907754] alg: No test for adler32 (adler32-zlib) [ 1543.109302] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [ 1543.367479] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 1545.090390] Key type lgssc registered [ 1546.485351] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1555.259801] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1555.276484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1556.685532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1561.050492] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1566.198203] LustreError: 62647:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1566.210905] LustreError: 62647:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1571.302574] LustreError: 62647:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1575.861394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1576.474289] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1582.504794] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1588.616696] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1592.762770] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1605.193862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1605.621732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1613.438690] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1620.509921] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1627.130839] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1637.345489] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1637.356753] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1637.365869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1637.367439] Lustre: Skipped 1 previous similar message [ 1637.381341] Lustre: Skipped 1 previous similar message [ 1642.466239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1642.474761] Lustre: Skipped 1 previous similar message [ 1642.557848] LustreError: 65301:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1642.771750] Lustre: server umount lustre-OST0000 complete [ 1652.708280] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1652.716771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1652.721897] Lustre: Skipped 1 previous similar message [ 1654.733583] LustreError: 65550:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1654.754357] LustreError: 65550:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1654.935624] Lustre: server umount lustre-MDT0000 complete [ 1657.825300] LustreError: 62642:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1657.842045] LustreError: 62642:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1660.083582] LustreError: 65551:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767316416 with bad export cookie 10527596311588959650 [ 1660.088532] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1660.090125] LustreError: 65551:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1660.215559] LustreError: 65804:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1660.220417] LustreError: 65804:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1660.349354] Lustre: server umount lustre-MDT0001 complete [ 1675.781499] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 1679.551578] Key type lgssc unregistered [ 1679.942246] LNet: 66481:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1679.949135] LNetError: 66481:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1679.982895] LNet: Removed LNI 192.168.204.105@tcp [ 1681.081937] Key type .llcrypt unregistered [ 1681.084698] Key type ._llcrypt unregistered [ 1768.508540] Key type ._llcrypt registered [ 1768.510048] Key type .llcrypt registered [ 1768.633124] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 1769.484730] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1769.525951] alg: No test for adler32 (adler32-zlib) [ 1770.561788] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [ 1770.772376] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 1772.439707] Key type lgssc registered [ 1773.570782] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1780.652396] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1780.683660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1782.122641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1786.536854] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1787.361369] LustreError: 70949:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1787.377933] LustreError: 70949:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1792.485517] LustreError: 70949:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1797.606937] LustreError: 70949:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1797.642087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1797.968389] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1801.728704] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1807.106460] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1810.882746] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1820.778027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1821.139211] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1826.923422] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 1827.273941] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1832.764857] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1842.680299] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1842.681544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1842.693427] Lustre: Skipped 2 previous similar messages [ 1842.716754] Lustre: Skipped 2 previous similar messages [ 1845.634039] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.5@tcp (stopping) [ 1847.018069] LustreError: 73629:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1847.136705] Lustre: server umount lustre-MDT0000 complete [ 1847.778210] LustreError: 70950:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1847.798466] LustreError: 70950:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1852.696554] LustreError: 70931:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767316608 with bad export cookie 8875499208357833664 [ 1852.721290] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1852.727326] LustreError: 70931:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1852.896930] LustreError: 70946:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1852.898701] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1852.916351] LustreError: 70946:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1852.947149] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1855.887189] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.5@tcp (stopping) [ 1858.973234] LustreError: 73882:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1858.987848] LustreError: 73882:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1859.187597] Lustre: server umount lustre-MDT0001 complete [ 1874.218484] LustreError: 74106:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1874.224682] LustreError: 74106:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1874.363703] Lustre: server umount lustre-OST0000 complete [ 1895.433115] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 1898.485303] Key type lgssc unregistered [ 1898.769443] LNet: 75184:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1898.782569] LNetError: 75184:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1898.811185] LNet: Removed LNI 192.168.204.105@tcp [ 1899.563401] Key type .llcrypt unregistered [ 1899.564933] Key type ._llcrypt unregistered [ 1979.718983] Key type ._llcrypt registered [ 1979.723579] Key type .llcrypt registered [ 1979.831702] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 1981.099886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1981.130893] alg: No test for adler32 (adler32-zlib) [ 1982.242729] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [ 1982.429623] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 1984.079343] Key type lgssc registered [ 1985.375756] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1992.178545] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1992.209344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1993.859329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1997.241569] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2004.459067] LustreError: 79642:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2004.483823] LustreError: 79642:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2009.112156] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2009.523375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2013.349416] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2018.651229] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2022.448274] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2031.871571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2032.163463] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2039.406179] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2040.365931] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 2045.733825] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2053.018250] Lustre: lustre-OST0000: Not available for connect from 192.168.204.5@tcp (stopping) [ 2055.651870] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2055.655663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2055.687243] Lustre: Skipped 1 previous similar message [ 2055.717844] Lustre: Skipped 1 previous similar message [ 2058.085580] Lustre: lustre-OST0000: Not available for connect from 192.168.204.5@tcp (stopping) [ 2058.641087] LustreError: 82276:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2058.803665] Lustre: server umount lustre-OST0000 complete [ 2080.242907] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2080.261491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2080.266450] Lustre: Skipped 1 previous similar message [ 2080.279550] Lustre: Skipped 1 previous similar message [ 2084.452114] LustreError: 82701:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2084.462719] LustreError: 82701:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2085.344657] LustreError: 80548:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2085.381803] LustreError: 80548:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2086.564070] LustreError: 82701:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 2086.573613] LustreError: 82701:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2086.705764] Lustre: server umount lustre-MDT0000 complete [ 2090.472322] LustreError: 80548:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2090.492813] LustreError: 80548:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2091.291609] LustreError: 82702:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767316847 with bad export cookie 12099757126658828370 [ 2091.294815] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2091.305351] LustreError: 82702:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2097.544903] LustreError: 82952:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2097.551557] LustreError: 82952:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2097.735313] Lustre: server umount lustre-MDT0001 complete [ 2110.618188] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 2113.003425] Key type lgssc unregistered [ 2113.294868] LNet: 83629:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2113.301666] LNetError: 83629:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2113.317968] LNet: Removed LNI 192.168.204.105@tcp [ 2114.094156] Key type .llcrypt unregistered [ 2114.097222] Key type ._llcrypt unregistered [ 2198.263891] Key type ._llcrypt registered [ 2198.266423] Key type .llcrypt registered [ 2198.389323] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 2199.960239] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2199.984308] alg: No test for adler32 (adler32-zlib) [ 2200.966789] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [ 2201.162801] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 2202.927461] Key type lgssc registered [ 2203.913866] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2210.522468] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2210.555540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2212.067949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2215.747192] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2222.563690] LustreError: 88088:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2222.580876] LustreError: 88088:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2227.687439] LustreError: 88088:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2228.212189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2228.557892] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2232.700070] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2238.606923] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2242.728428] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2252.945878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2253.225026] LustreError: 90072:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2253.403047] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2257.545572] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 2258.072454] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2262.659882] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2267.850304] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 2271.431252] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 2283.310964] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-01-0120h23m56s [ 2284.854172] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 2288.622227] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2288.633600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2288.645621] Lustre: Skipped 1 previous similar message [ 2288.668610] Lustre: Skipped 3 previous similar messages [ 2289.706110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2289.714310] Lustre: Skipped 2 previous similar messages [ 2291.113363] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.5@tcp (stopping) [ 2291.126322] Lustre: Skipped 1 previous similar message [ 2293.649509] LustreError: 91089:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2295.715467] LustreError: 91089:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 2295.717780] LustreError: 91089:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2295.756937] Lustre: server umount lustre-MDT0000 complete [ 2295.787459] LustreError: 88083:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2295.815146] LustreError: 88083:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2299.465881] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 2301.941780] LustreError: 88993:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2301.956258] LustreError: 88993:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2309.522036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2309.606157] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2309.833511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2309.920758] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 2310.829967] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2313.543862] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2316.601991] Lustre: DEBUG MARKER: Wait for df (44533) ... [ 2318.227227] Lustre: DEBUG MARKER: done [ 2324.449520] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2324.452228] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2324.480652] Lustre: Skipped 1 previous similar message [ 2324.486467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2325.473086] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2325.482264] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2331.626857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2331.636941] Lustre: Skipped 4 previous similar messages [ 2336.223247] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2336.267335] LustreError: 92337:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2336.273902] LustreError: 92337:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 2336.547932] Lustre: server umount lustre-OST0000 complete [ 2341.131490] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2341.158588] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2341.164093] Lustre: Skipped 1 previous similar message [ 2347.423720] LustreError: 92589:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2347.432381] LustreError: 92589:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2347.670439] Lustre: server umount lustre-MDT0000 complete [ 2348.514202] LustreError: 88089:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2348.527573] LustreError: 88089:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 2353.355939] LustreError: 90768:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767317109 with bad export cookie 10329040180990051621 [ 2353.364229] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2353.366349] LustreError: 90768:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2353.635904] LustreError: 92839:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2353.643219] LustreError: 92839:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2354.000400] Lustre: server umount lustre-MDT0001 complete [ 2366.537171] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 2369.427882] Key type lgssc unregistered [ 2369.846809] LNet: 93515:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2369.854970] LNetError: 93515:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2369.885679] LNet: Removed LNI 192.168.204.105@tcp [ 2370.912984] Key type .llcrypt unregistered [ 2370.914738] Key type ._llcrypt unregistered [ 2419.695165] Key type ._llcrypt registered [ 2419.697187] Key type .llcrypt registered [ 2419.798988] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 2482.960866] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 2484.009184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2484.074157] alg: No test for adler32 (adler32-zlib) [ 2485.089046] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [ 2485.260241] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 2486.939517] Key type lgssc registered [ 2488.080233] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2503.510024] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2504.112710] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2505.477169] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2507.570696] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2520.785778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2520.855397] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2520.866991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2522.197938] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2522.226821] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2522.294755] Lustre: lustre-MDT0000: new disk, initializing [ 2522.373556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2522.391729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2526.177990] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2540.852186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2540.914934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2541.004710] Lustre: 103547:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2541.050784] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2541.054911] Lustre: Skipped 1 previous similar message [ 2541.131204] Lustre: lustre-MDT0001: new disk, initializing [ 2541.231359] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2541.290139] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2541.306429] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2545.315735] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2553.761462] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2557.651897] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2572.038693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2572.112725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2572.320111] Lustre: lustre-OST0000: new disk, initializing [ 2572.328294] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2572.407986] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2578.515961] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2582.061081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2582.074373] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2582.213732] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2586.288528] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2590.948731] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2591.178944] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2595.706871] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2595.923370] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2597.859879] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2597.870284] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2597.884742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2597.889633] Lustre: Skipped 1 previous similar message [ 2602.980420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2602.984273] Lustre: Skipped 1 previous similar message [ 2608.106118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2608.114054] Lustre: Skipped 1 previous similar message [ 2612.191197] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2612.241167] LustreError: 107178:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2612.570420] Lustre: server umount lustre-OST0000 complete [ 2618.343736] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2618.346743] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2618.367825] Lustre: Skipped 1 previous similar message [ 2618.369097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2618.400643] LustreError: Skipped 1 previous similar message [ 2623.183826] LustreError: 107430:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2623.223486] LustreError: 107430:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2623.357183] Lustre: server umount lustre-MDT0000 complete [ 2627.604982] LustreError: 104682:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767317383 with bad export cookie 6930718459845604329 [ 2627.615815] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2627.738661] LustreError: 107680:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2627.747219] LustreError: 107680:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2627.933241] Lustre: server umount lustre-MDT0001 complete [ 2635.680742] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 20:29:49 (1767317389) [ 2645.164297] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2645.467650] LustreError: 108543:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2645.481510] LustreError: 108543:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2645.524240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2650.061980] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2650.597490] LustreError: 108543:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2655.726729] LustreError: 108543:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2660.836546] LustreError: 108543:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2662.300797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2662.554287] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2666.423871] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2672.294724] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2675.970373] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2683.835858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2684.189111] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2690.592476] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2695.175712] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2701.694924] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-01-0120h30m55s [ 2703.512339] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 2712.005734] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 2714.811782] Lustre: DEBUG MARKER: Creating a test file and stat it [ 2714.855472] LustreError: 108537:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 2734.991315] Lustre: lustre-MDT0000: Client ef75f32b-4c93-486a-ae3c-a01b8f144ade (at 192.168.204.5@tcp) reconnecting [ 2754.918874] LustreError: 108537:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 2757.440850] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 2759.708958] Lustre: DEBUG MARKER: done [ 2766.936214] LustreError: 111932:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2766.945444] LustreError: 111932:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2767.112417] Lustre: server umount lustre-OST0000 complete [ 2772.451105] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2772.452729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2772.472349] Lustre: Skipped 2 previous similar messages [ 2772.481798] Lustre: Skipped 1 previous similar message [ 2781.706306] LustreError: 112181:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2781.716975] LustreError: 112181:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2781.828466] Lustre: server umount lustre-MDT0000 complete [ 2783.713406] Lustre: 100021:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317523/real 1767317523] req@0000000005acf496 x1853166443391616/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1767317539 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2783.765380] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2784.815481] LustreError: 109448:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2785.824622] Lustre: 100023:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317525/real 1767317525] req@00000000f1af4fb9 x1853166443391872/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1767317541 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2787.394533] LustreError: 108523:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767317543 with bad export cookie 6930718459845605211 [ 2787.400715] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2787.410340] LustreError: 108523:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 2787.833457] Lustre: server umount lustre-MDT0001 complete [ 2801.527402] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 2803.840137] Key type lgssc unregistered [ 2804.074486] LNet: 113109:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2804.079842] LNetError: 113109:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2805.094189] LNet: Removed LNI 192.168.204.105@tcp [ 2805.746840] Key type .llcrypt unregistered [ 2805.751841] Key type ._llcrypt unregistered [ 2845.198739] Key type ._llcrypt registered [ 2845.200357] Key type .llcrypt registered [ 2845.263926] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 2903.101721] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 2904.168852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2904.226289] alg: No test for adler32 (adler32-zlib) [ 2905.214075] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [ 2905.364897] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 2907.002644] Key type lgssc registered [ 2908.051922] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2923.391456] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2924.063988] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2925.417144] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2928.035839] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2942.049452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2942.102387] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2942.116843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2943.397282] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2943.428162] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2943.533560] Lustre: lustre-MDT0000: new disk, initializing [ 2943.647901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2943.671995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2948.126336] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2966.555153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2966.671388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2966.789829] Lustre: 123140:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2966.880423] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2966.885912] Lustre: Skipped 1 previous similar message [ 2966.991707] Lustre: lustre-MDT0001: new disk, initializing [ 2967.131649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2967.202399] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2967.225220] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2970.968828] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 2977.797651] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2981.867469] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2992.820248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2992.876264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2993.043147] Lustre: lustre-OST0000: new disk, initializing [ 2993.048658] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2993.162562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2998.734205] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3003.487851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3003.497938] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3003.578362] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3006.709610] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3011.665666] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3011.860533] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3016.667308] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3016.917912] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3019.238947] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3019.250460] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3019.280551] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3019.287346] Lustre: Skipped 1 previous similar message [ 3024.359605] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3024.374504] Lustre: Skipped 1 previous similar message [ 3024.792267] LustreError: 126772:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3024.887466] Lustre: server umount lustre-OST0000 complete [ 3029.473699] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3029.478560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3029.488409] Lustre: Skipped 2 previous similar messages [ 3029.505465] Lustre: Skipped 1 previous similar message [ 3034.593619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3034.600482] Lustre: Skipped 1 previous similar message [ 3039.718552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3039.724255] Lustre: Skipped 1 previous similar message [ 3043.295183] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3043.380273] LustreError: 127022:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3043.384530] LustreError: 127022:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3043.473987] Lustre: server umount lustre-MDT0000 complete [ 3046.971759] LustreError: 123135:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767317803 with bad export cookie 5314393727169247621 [ 3046.983519] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3047.074565] LustreError: 127273:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3047.088164] LustreError: 127273:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3047.197159] Lustre: server umount lustre-MDT0001 complete [ 3054.336819] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 20:36:48 (1767317808) [ 3055.669726] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 3057.156814] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 20:36:51 (1767317811) [ 3112.784740] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 3116.272858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3116.607657] LustreError: 131257:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3116.620831] LustreError: 131257:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3116.660161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3117.486281] Lustre: Failing over lustre-MDT0000 [ 3117.495784] LustreError: 131285:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3117.503959] LustreError: 131285:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3117.510348] LustreError: 131285:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 1, retries 0, failed: rc = -5 [ 3117.616980] LustreError: 131336:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3117.621026] LustreError: 131336:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3117.807759] Lustre: server umount lustre-MDT0000 complete [ 3122.936031] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 20:37:57 (1767317877) [ 3131.768536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3132.078303] LustreError: 132095:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3132.116211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3135.316472] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3137.512095] LustreError: 132113:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3142.624229] LustreError: 132113:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3144.064507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3144.285792] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3147.398842] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3151.978624] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3154.945261] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3163.340813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3163.613235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3168.404588] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3172.605152] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3182.723191] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 3189.214683] LustreError: 119616:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000297dc9ba x1853166883771904/t0(0) o2->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3189.332129] LustreError: 134872:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3189.339375] LustreError: 134872:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3189.439102] Lustre: server umount lustre-MDT0000 complete [ 3190.242739] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3190.254379] LustreError: Skipped 1 previous similar message [ 3190.259263] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3190.274689] LustreError: 132113:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3192.288626] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3193.650767] LustreError: 134098:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767317949 with bad export cookie 5314393727169248776 [ 3193.658395] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3193.667037] LustreError: 134098:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3193.960948] Lustre: server umount lustre-MDT0001 complete [ 3196.835719] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 3201.026415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3212.583184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3213.791503] Lustre: 119614:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317953/real 1767317953] req@000000001253b93a x1853166883774848/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1767317969 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3213.828042] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3213.837216] Lustre: Skipped 1 previous similar message [ 3214.815244] Lustre: 119616:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317954/real 1767317954] req@00000000cbe8430b x1853166883774976/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1767317970 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3217.951295] Lustre: 119616:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317958/real 1767317958] req@00000000e420d1a4 x1853166883775104/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1767317974 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3218.291769] LustreError: 134087:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3218.306935] LustreError: 134087:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3218.383972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3218.464263] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 3222.157643] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3224.479104] Lustre: 119614:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317964/real 1767317964] req@000000007cd7a009 x1853166883775488/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1767317980 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3224.493098] Lustre: 119614:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3228.639424] Lustre: 119617:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767317968/real 1767317968] req@00000000f59ab754 x1853166883775744/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1767317984 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3232.456315] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3233.781819] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3237.140847] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3242.996779] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3247.255284] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3253.672469] LustreError: 137920:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3253.679745] LustreError: 137920:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 3254.241846] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3254.245228] LustreError: 137010:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3254.267627] Lustre: Skipped 2 previous similar messages [ 3254.290647] LustreError: 137010:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 3255.866297] Lustre: server umount lustre-MDT0000 complete [ 3260.691893] LustreError: 134098:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767318016 with bad export cookie 5314393727169250680 [ 3260.695031] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3260.700269] LustreError: 134098:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3260.996154] Lustre: server umount lustre-MDT0001 complete [ 3266.325460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3279.839156] Lustre: 119614:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767318020/real 1767318020] req@0000000085d029ee x1853166883801216/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1767318036 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3279.858265] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3280.058132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3285.024476] LustreError: 119613:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000059817db8 x1853166883802240/t0(0) o250->MGC192.168.204.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3285.311624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3285.318540] Lustre: Skipped 1 previous similar message [ 3285.369612] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 3288.356104] LustreError: 134086:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3288.386889] LustreError: 134086:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 3289.881774] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3306.105335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3309.612424] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 3309.620536] Lustre: Skipped 1 previous similar message [ 3312.048417] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3317.409982] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3321.870227] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3332.069494] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3332.088854] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3332.111840] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3336.925552] LustreError: 140958:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3336.941040] LustreError: 140958:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 3337.102282] Lustre: server umount lustre-MDT0000 complete [ 3341.584366] LustreError: 132075:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767318097 with bad export cookie 5314393727169252052 [ 3341.590688] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3341.600439] LustreError: 132075:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3341.854907] Lustre: server umount lustre-MDT0001 complete [ 3348.708345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 3352.669680] Lustre: DEBUG MARKER: files compared the same [ 3364.999618] Lustre: server umount lustre-OST0000 complete [ 3381.810623] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 3383.865663] Key type lgssc unregistered [ 3384.055745] LNet: 142977:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3384.059459] LNetError: 142977:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3384.080939] LNet: Removed LNI 192.168.204.105@tcp [ 3384.545920] Key type .llcrypt unregistered [ 3384.547386] Key type ._llcrypt unregistered [ 3456.408701] Key type ._llcrypt registered [ 3456.410810] Key type .llcrypt registered [ 3456.487502] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 3457.361074] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3457.396778] alg: No test for adler32 (adler32-zlib) [ 3458.462811] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [ 3458.668252] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 3460.319218] Key type lgssc registered [ 3461.252881] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3467.861753] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3467.891551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3469.284778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3473.145440] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug +malloc all [ 3479.524468] LustreError: 147432:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3479.543615] LustreError: 147432:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3484.166354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3484.434821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3488.258262] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug +malloc all [ 3493.471762] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3497.087478] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3507.353879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3507.793404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3513.866714] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug +malloc all [ 3513.977169] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 3519.461966] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3529.695777] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3529.704885] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3529.733724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3529.746220] Lustre: Skipped 1 previous similar message [ 3533.883432] LustreError: 150064:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3534.003596] Lustre: server umount lustre-OST0000 complete [ 3539.936786] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3539.937525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3539.951385] Lustre: Skipped 2 previous similar messages [ 3539.973960] Lustre: Skipped 1 previous similar message [ 3544.439778] LustreError: 150315:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3544.448487] LustreError: 150315:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3544.509745] Lustre: server umount lustre-MDT0000 complete [ 3545.056732] LustreError: 149280:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3545.073575] LustreError: 149280:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3548.657919] LustreError: 147412:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767318304 with bad export cookie 11851804736895467308 [ 3548.662679] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3548.665169] LustreError: 147412:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3548.789060] LustreError: 150565:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3548.794146] LustreError: 150565:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3548.959468] Lustre: server umount lustre-MDT0001 complete [ 3560.162227] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 3562.291096] Key type lgssc unregistered [ 3562.527177] LNet: 151241:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3562.542462] LNetError: 151241:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3563.567636] LNet: Removed LNI 192.168.204.105@tcp [ 3564.316996] Key type .llcrypt unregistered [ 3564.318828] Key type ._llcrypt unregistered [ 3631.322403] Key type ._llcrypt registered [ 3631.324945] Key type .llcrypt registered [ 3631.418936] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 3632.101782] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3632.168037] alg: No test for adler32 (adler32-zlib) [ 3633.241707] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [ 3633.462224] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 3635.111978] Key type lgssc registered [ 3636.004968] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3642.794770] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3642.808797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3658.400540] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3663.519467] LustreError: 155670:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.105@tcp: failed processing log, type 4: rc = -110 [ 3690.271181] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3695.841187] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3700.900232] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3711.114773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3711.237377] Lustre: *** cfs_fail_loc=706, val=0*** [ 3711.587569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3711.637647] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 3715.480270] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3717.089467] LustreError: 155697:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3717.119775] LustreError: 155697:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3722.214418] LustreError: 155697:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3725.294065] LustreError: 156836:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3725.336330] LustreError: 156836:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3727.944248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3728.228569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3732.925265] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3738.571569] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3742.389991] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3747.302405] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3747.306064] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3747.328631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3748.840258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3748.841146] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3748.856592] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3750.593201] LustreError: 158381:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 3750.735987] Lustre: server umount lustre-OST0000 complete [ 3759.072433] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3759.089416] Lustre: Skipped 1 previous similar message [ 3759.092875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3759.101662] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3759.130653] Lustre: Skipped 2 previous similar messages [ 3761.569116] LustreError: 158630:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 3761.573904] LustreError: 158630:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3761.631298] Lustre: server umount lustre-MDT0000 complete [ 3764.193397] LustreError: 156836:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3764.209745] LustreError: 156836:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3766.421185] LustreError: 155678:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767318522 with bad export cookie 15492844662819769367 [ 3766.423048] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3766.428465] LustreError: 155678:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3766.553362] LustreError: 158881:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3766.557545] LustreError: 158881:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3766.776490] Lustre: server umount lustre-MDT0001 complete [ 3779.879860] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 3782.213687] Key type lgssc unregistered [ 3782.517791] LNet: 159560:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3782.526577] LNetError: 159560:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3782.563897] LNet: Removed LNI 192.168.204.105@tcp [ 3783.466791] Key type .llcrypt unregistered [ 3783.469895] Key type ._llcrypt unregistered [ 3867.476858] Key type ._llcrypt registered [ 3867.479960] Key type .llcrypt registered [ 3867.570817] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 3868.766609] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3868.823110] alg: No test for adler32 (adler32-zlib) [ 3869.955671] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [ 3870.242254] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 3871.919373] Key type lgssc registered [ 3873.183866] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3881.002358] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3881.021035] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3886.320340] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3897.407289] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3898.371248] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3902.291445] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3903.456029] LustreError: 165087:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3903.484288] LustreError: 165087:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3908.579099] LustreError: 165092:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3908.604517] LustreError: 165092:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3913.702145] LustreError: 165092:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3913.732583] LustreError: 165092:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3915.142128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3915.694677] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3920.866974] LustreError: 165088:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3920.897261] LustreError: 165088:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3922.810643] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3928.039820] LustreError: 165092:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3928.063223] LustreError: 165092:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3935.951159] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3935.960533] Lustre: MGS: Client de122158-d75f-41c8-b558-63ef3901cdba (at 0@lo) reconnecting [ 3936.278845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3936.385147] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 3940.068586] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 3956.706076] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3956.716036] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3956.719899] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3956.723014] Lustre: Skipped 1 previous similar message [ 3956.745605] Lustre: Skipped 1 previous similar message [ 3958.984124] LustreError: 167437:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 3959.210690] Lustre: server umount lustre-OST0000 complete [ 3966.944154] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3966.946162] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3966.952653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3966.979658] Lustre: Skipped 1 previous similar message [ 3969.519920] LustreError: 167687:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3969.535454] LustreError: 167687:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3969.577984] Lustre: server umount lustre-MDT0000 complete [ 3972.074278] LustreError: 165086:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3972.088770] LustreError: 165086:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3973.653800] LustreError: 167939:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3973.659433] LustreError: 167939:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3973.801396] Lustre: server umount lustre-MDT0001 complete [ 3977.963833] LustreError: 168188:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3977.970824] LustreError: 168188:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3978.038381] Lustre: server umount MGS complete [ 3990.424925] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 20:52:24 (1767318744) [ 4026.509191] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 4095.789764] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 4117.025144] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4117.052029] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4118.685282] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4120.552162] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4132.383670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4132.438881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4136.272595] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4150.104310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4150.184545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4150.322908] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 4150.360789] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4150.428655] Lustre: lustre-MDT0001: new disk, initializing [ 4150.499279] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4150.530937] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 4150.541490] Lustre: 179394:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 4150.553604] LustreError: 179394:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 4154.447715] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4171.430622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4171.514092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4171.861253] Lustre: lustre-OST0000: new disk, initializing [ 4171.867388] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4171.951507] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4177.216332] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4181.603202] Lustre: 180965:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@00000000b5317431 x1853167895618688/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1767318953 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 4184.723320] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4195.670708] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4195.687633] Lustre: MGS: Client 5e0d0c36-414f-49c3-8db7-ff42814875c4 (at 0@lo) reconnecting [ 4195.785283] Lustre: 178334:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4195.822319] LustreError: 178334:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 4195.868858] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4195.968213] Lustre: lustre-MDT0000: new disk, initializing [ 4196.204290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4196.241747] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4196.282921] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 4197.358309] Lustre: 181646:0:(client.c:1637:after_reply()) @@@ resending request on EINPROGRESS req@000000000fcda79c x1853167895625216/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1767318969 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 4202.275961] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4202.478128] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 4206.492995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4206.504582] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4206.559345] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 4229.604738] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4229.612207] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4229.630995] Lustre: Skipped 1 previous similar message [ 4229.636602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4231.665823] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4231.688979] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4233.670462] LustreError: 182168:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 4233.675701] LustreError: 182168:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 4233.948456] Lustre: server umount lustre-OST0000 complete [ 4241.891329] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4241.892368] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4241.905089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4241.915575] Lustre: Skipped 2 previous similar messages [ 4244.080413] LustreError: 182418:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 4244.084195] LustreError: 182418:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4244.100860] Lustre: server umount lustre-MDT0000 complete [ 4247.009778] LustreError: 179369:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4247.032211] LustreError: 179369:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4262.370158] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4262.488594] LustreError: 182668:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 4262.492134] LustreError: 182668:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4262.594119] Lustre: server umount lustre-MDT0001 complete [ 4266.443489] Lustre: server umount MGS complete [ 4276.213782] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 20:57:10 (1767319030) [ 4295.655816] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 4298.032092] Key type lgssc unregistered [ 4298.376779] LNet: 184543:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4298.383591] LNetError: 184543:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4298.397561] LNet: Removed LNI 192.168.204.105@tcp [ 4299.288644] Key type .llcrypt unregistered [ 4299.291870] Key type ._llcrypt unregistered [ 4364.092341] Key type ._llcrypt registered [ 4364.094535] Key type .llcrypt registered [ 4364.185640] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 4364.738312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4364.812132] alg: No test for adler32 (adler32-zlib) [ 4365.934084] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [ 4366.150600] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 4367.799238] Key type lgssc registered [ 4368.815812] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4375.288211] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4375.308675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4376.838810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4381.624957] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4387.296475] LustreError: 188803:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4387.305322] LustreError: 188803:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4392.425886] LustreError: 188803:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4393.097241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4393.355381] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4396.643558] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4401.764485] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4405.069428] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4413.746302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4414.079893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4419.635535] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4422.284612] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 4425.069941] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4432.869172] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4432.876257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4432.880752] Lustre: Skipped 1 previous similar message [ 4432.881543] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4432.917935] Lustre: Skipped 1 previous similar message [ 4437.252386] LustreError: 191434:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4437.487509] Lustre: server umount lustre-OST0000 complete [ 4443.105422] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4443.112479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4443.114683] Lustre: Skipped 1 previous similar message [ 4448.059952] LustreError: 191684:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4448.067408] LustreError: 191684:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4448.221521] Lustre: server umount lustre-MDT0000 complete [ 4448.224200] LustreError: 188803:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4448.270845] LustreError: 188803:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4452.191568] LustreError: 188784:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767319208 with bad export cookie 2069425285988449239 [ 4452.192891] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4452.205323] LustreError: 188784:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4452.332595] LustreError: 191936:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 4452.338534] LustreError: 191936:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4452.476085] Lustre: server umount lustre-MDT0001 complete [ 4467.944122] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 4470.897917] Key type lgssc unregistered [ 4471.330966] LNet: 192612:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4471.344713] LNetError: 192612:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4471.387259] LNet: Removed LNI 192.168.204.105@tcp [ 4472.267270] Key type .llcrypt unregistered [ 4472.270075] Key type ._llcrypt unregistered [ 4538.071105] Key type ._llcrypt registered [ 4538.073265] Key type .llcrypt registered [ 4538.208880] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 4539.168782] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4539.374673] alg: No test for adler32 (adler32-zlib) [ 4540.342695] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [ 4540.619948] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 4542.224196] Key type lgssc registered [ 4543.150680] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4548.787740] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4548.793500] LustreError: 196597:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 4548.823842] LustreError: 196605:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 4548.827968] LustreError: 196597:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4970 [ 4548.831730] LustreError: 196597:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 4548.837269] LustreError: 196597:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 4548.844303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4550.248248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4559.855509] LustreError: 196643:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4559.886582] LustreError: 196643:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4560.688411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4561.086433] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4564.533795] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4569.073458] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing lsmod [ 4571.976916] LustreError: 198010:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 4572.022625] LustreError: 198023:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 4572.027858] LustreError: 198010:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4954 [ 4572.036199] LustreError: 198010:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 4572.042681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4572.049071] LustreError: 198010:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 4572.386360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4576.736972] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4576.746936] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4576.756660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4577.767659] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4577.775618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4577.788783] Lustre: Skipped 1 previous similar message [ 4577.793983] Lustre: Skipped 1 previous similar message [ 4581.539852] LustreError: 198258:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4581.666426] Lustre: server umount lustre-MDT0000 complete [ 4582.882668] LustreError: 196643:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4582.897409] LustreError: 196643:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4585.607914] LustreError: 198062:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767319341 with bad export cookie 7227085644534388702 [ 4585.610549] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4585.614113] LustreError: 198062:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4585.722420] LustreError: 198509:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 4585.726340] LustreError: 198509:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4585.921537] Lustre: server umount lustre-MDT0001 complete [ 4598.971705] LustreError: 198710:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4598.976560] LustreError: 198710:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4599.116685] Lustre: server umount lustre-OST0000 complete [ 4609.678614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4610.370682] LustreError: 199380:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4610.491662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4615.519150] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4615.658677] LustreError: 199380:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4620.773494] LustreError: 199380:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4628.626978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4628.941221] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4632.358691] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4637.010625] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4639.859978] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4649.829735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4650.301193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4655.377903] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4655.435745] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 4660.214607] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4669.921831] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4669.930198] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4669.947337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4669.964848] Lustre: Skipped 1 previous similar message [ 4670.948834] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4672.458480] LustreError: 202009:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4672.472147] LustreError: 202009:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4672.626864] Lustre: server umount lustre-OST0000 complete [ 4680.161652] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4680.169361] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4680.181234] Lustre: Skipped 1 previous similar message [ 4680.191038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4680.195537] Lustre: Skipped 2 previous similar messages [ 4682.663527] LustreError: 202260:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4682.676768] LustreError: 202260:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4682.782144] Lustre: server umount lustre-MDT0000 complete [ 4686.304540] LustreError: 199375:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4686.327030] LustreError: 199375:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4686.938324] LustreError: 202010:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767319443 with bad export cookie 7227085644534389395 [ 4686.946052] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4686.950233] LustreError: 202010:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4687.248675] Lustre: server umount lustre-MDT0001 complete [ 4697.644886] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 4699.742856] Key type lgssc unregistered [ 4699.975736] LNet: 203185:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4699.986056] LNetError: 203185:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4699.997432] LNet: Removed LNI 192.168.204.105@tcp [ 4700.726649] Key type .llcrypt unregistered [ 4700.728844] Key type ._llcrypt unregistered [ 4772.347627] Key type ._llcrypt registered [ 4772.357256] Key type .llcrypt registered [ 4772.438941] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 4773.579227] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4773.610535] alg: No test for adler32 (adler32-zlib) [ 4774.726918] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [ 4775.008116] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 4776.711264] Key type lgssc registered [ 4777.646815] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4784.587379] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4784.610498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4786.056838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4790.430733] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4796.384370] LustreError: 207642:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4796.400524] LustreError: 207642:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4800.085774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4800.412971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4804.150335] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4808.748623] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4812.006311] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4820.828634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4821.193347] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4826.187071] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4830.238731] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4831.412487] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 4836.023192] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 4839.109983] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 4846.052284] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4846.063891] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4846.071954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4847.079824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4847.085228] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4847.088446] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4849.847106] LustreError: 210381:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4849.990508] Lustre: server umount lustre-OST0000 complete [ 4856.801699] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4856.814639] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4856.829161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4856.831459] Lustre: Skipped 1 previous similar message [ 4860.456353] LustreError: 210631:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4860.464610] LustreError: 210631:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4860.814737] Lustre: server umount lustre-MDT0000 complete [ 4862.433348] LustreError: 207642:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4862.449782] LustreError: 207642:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4864.983700] LustreError: 207623:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767319621 with bad export cookie 6311280856020701875 [ 4864.987984] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4864.995424] LustreError: 207623:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4865.175710] LustreError: 210882:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 4865.181920] LustreError: 210882:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4865.542376] Lustre: server umount lustre-MDT0001 complete [ 4878.997471] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 4881.645837] Key type lgssc unregistered [ 4881.932801] LNet: 211558:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4881.939918] LNetError: 211558:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4881.954336] LNet: Removed LNI 192.168.204.105@tcp [ 4882.851956] Key type .llcrypt unregistered [ 4882.853907] Key type ._llcrypt unregistered [ 4946.287429] Key type ._llcrypt registered [ 4946.292363] Key type .llcrypt registered [ 4946.385779] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 4947.176104] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4947.216659] alg: No test for adler32 (adler32-zlib) [ 4948.211320] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [ 4948.383222] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 4950.031526] Key type lgssc registered [ 4951.006382] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4956.671249] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4956.684506] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4958.146636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4961.469296] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4968.419797] LustreError: 215818:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4968.429643] LustreError: 215818:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4972.564953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4972.815217] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4976.285934] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 4980.742159] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4984.174996] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4994.267488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4994.566772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4999.606600] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5003.836237] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 5005.386669] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5013.986386] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5013.990231] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5013.996358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5014.007045] Lustre: Skipped 1 previous similar message [ 5014.020061] Lustre: Skipped 1 previous similar message [ 5019.107920] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5019.112464] Lustre: Skipped 1 previous similar message [ 5019.602140] LustreError: 218448:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5019.733837] Lustre: server umount lustre-OST0000 complete [ 5024.225031] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5024.234976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5024.241898] Lustre: Skipped 1 previous similar message [ 5029.346448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5029.351636] Lustre: Skipped 2 previous similar messages [ 5029.543942] LustreError: 218697:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5029.552356] LustreError: 218697:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5029.624828] Lustre: server umount lustre-MDT0000 complete [ 5034.230717] LustreError: 215800:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767319790 with bad export cookie 2751154548929813760 [ 5034.233571] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5034.241040] LustreError: 215800:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5034.367973] LustreError: 218947:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 5034.371577] LustreError: 218947:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5034.503288] Lustre: server umount lustre-MDT0001 complete [ 5048.439517] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 5050.828864] Key type lgssc unregistered [ 5051.113266] LNet: 219626:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5051.122926] LNetError: 219626:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5051.134178] LNet: Removed LNI 192.168.204.105@tcp [ 5051.885258] Key type .llcrypt unregistered [ 5051.894100] Key type ._llcrypt unregistered [ 5125.027608] Key type ._llcrypt registered [ 5125.029768] Key type .llcrypt registered [ 5125.132069] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 5125.849785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5125.875459] alg: No test for adler32 (adler32-zlib) [ 5126.890331] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [ 5127.051046] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 5128.680581] Key type lgssc registered [ 5129.744515] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5136.496868] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5136.522688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5137.907740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5141.808784] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5148.131479] LustreError: 224082:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5148.160662] LustreError: 224082:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5153.253278] LustreError: 224082:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5154.306841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5154.626585] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5158.259423] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5164.325376] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5168.184616] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5178.943417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5179.240898] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5183.361073] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 5184.414761] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5189.286977] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5196.486893] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 5200.022240] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 5200.030416] Lustre: Skipped 2 previous similar messages [ 5204.075709] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 5204.083355] Lustre: Skipped 2 previous similar messages [ 5204.561565] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 5204.596096] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 5210.262359] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.5@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 5210.269371] Lustre: Skipped 2 previous similar messages [ 5211.372691] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.5@tcp 192.168.0.[2,10]@tcp [ 5217.248476] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.5/24@tcp in log lustre-MDT0000 [ 5217.254359] Lustre: Skipped 2 previous similar messages [ 5217.713190] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.5/24@tcp [ 5217.716309] Lustre: Skipped 1 previous similar message [ 5226.488293] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5226.498251] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5226.516576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5229.550082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5229.556778] Lustre: Skipped 2 previous similar messages [ 5231.673629] LustreError: 227780:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5231.839672] Lustre: server umount lustre-OST0000 complete [ 5236.703941] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5236.706345] LustreError: Skipped 6 previous similar messages [ 5236.711667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5236.724299] Lustre: Skipped 1 previous similar message [ 5236.726804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5239.777147] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5239.777918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5239.807824] Lustre: Skipped 1 previous similar message [ 5242.008233] LustreError: 228030:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5242.017189] LustreError: 228030:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5242.058803] Lustre: server umount lustre-MDT0000 complete [ 5244.898472] LustreError: 224083:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5244.921295] LustreError: 224083:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5245.439005] LustreError: 226767:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767320001 with bad export cookie 11311311674666578306 [ 5245.443019] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5245.450803] LustreError: 226767:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5245.576460] LustreError: 228279:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 5245.583268] LustreError: 228279:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5245.698826] Lustre: server umount lustre-MDT0001 complete [ 5256.946669] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 5259.020935] Key type lgssc unregistered [ 5259.269943] LNet: 228955:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5259.280291] LNetError: 228955:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5260.325485] LNet: Removed LNI 192.168.204.105@tcp [ 5261.015533] Key type .llcrypt unregistered [ 5261.018954] Key type ._llcrypt unregistered [ 5338.883226] Key type ._llcrypt registered [ 5338.885621] Key type .llcrypt registered [ 5338.955290] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 5339.816374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5339.926296] alg: No test for adler32 (adler32-zlib) [ 5340.972301] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [ 5341.192977] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 5342.863197] Key type lgssc registered [ 5343.758248] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5349.719747] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5349.735944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5351.057885] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 5351.089596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5351.100044] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.5/24@tcp [ 5354.985251] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5361.633529] LustreError: 233605:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5361.645345] LustreError: 233605:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5365.932954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5366.159051] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 5366.176594] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.5/24@tcp [ 5366.207866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5369.723489] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5374.619627] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5377.926986] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5386.115550] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5386.409575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5392.067653] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5396.661025] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 5397.242345] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5407.206800] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5407.210926] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5407.214198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5407.228337] Lustre: Skipped 1 previous similar message [ 5407.239833] Lustre: Skipped 1 previous similar message [ 5411.132187] LustreError: 236293:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5411.294623] Lustre: server umount lustre-OST0000 complete [ 5417.443160] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5417.446927] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5417.465463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5417.480736] Lustre: Skipped 1 previous similar message [ 5422.220208] LustreError: 236544:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5422.222814] LustreError: 236544:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5422.488051] Lustre: server umount lustre-MDT0000 complete [ 5422.561366] LustreError: 234847:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5422.572771] LustreError: 234847:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5426.658300] LustreError: 233586:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767320182 with bad export cookie 8300934987977677871 [ 5426.668357] LustreError: 233586:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5426.671415] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5426.804175] LustreError: 236793:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 5426.811509] LustreError: 236793:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5426.970858] Lustre: server umount lustre-MDT0001 complete [ 5439.966662] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 5442.642720] Key type lgssc unregistered [ 5442.917575] LNet: 237470:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5442.921306] LNetError: 237470:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5442.935188] LNet: Removed LNI 192.168.204.105@tcp [ 5443.613213] Key type .llcrypt unregistered [ 5443.615309] Key type ._llcrypt unregistered [ 5482.860445] Key type ._llcrypt registered [ 5482.862635] Key type .llcrypt registered [ 5482.992912] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 5550.142333] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 5551.203552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5551.230212] alg: No test for adler32 (adler32-zlib) [ 5552.257958] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [ 5552.511082] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 5554.209117] Key type lgssc registered [ 5555.347348] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5570.868797] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5571.249187] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5573.379554] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5574.949174] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5590.484477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5590.554295] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5590.577507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5591.932273] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5592.028542] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5592.205840] Lustre: lustre-MDT0000: new disk, initializing [ 5592.401545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5592.423381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5598.043869] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5613.954048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5614.014021] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5614.094644] Lustre: 246738:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5614.152697] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5614.155914] Lustre: Skipped 1 previous similar message [ 5614.215798] Lustre: lustre-MDT0001: new disk, initializing [ 5614.296871] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5614.329249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5614.344357] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5618.772898] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5628.688211] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5632.796619] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5644.669645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5644.753183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5644.926916] Lustre: lustre-OST0000: new disk, initializing [ 5644.933384] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5645.081450] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5650.921875] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5653.033824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5653.040756] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5653.128941] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5658.355689] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5663.303683] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5663.585608] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5667.950867] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5668.165847] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5673.452504] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5673.457109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5673.469546] Lustre: Skipped 1 previous similar message [ 5673.481831] Lustre: Skipped 1 previous similar message [ 5675.763362] LustreError: 250370:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5675.963951] Lustre: server umount lustre-OST0000 complete [ 5681.120616] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5681.125547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5681.131691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5683.683840] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5683.686673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5683.702266] Lustre: Skipped 1 previous similar message [ 5686.143208] LustreError: 250620:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5686.150445] LustreError: 250620:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5686.268133] Lustre: server umount lustre-MDT0000 complete [ 5690.677492] LustreError: 246727:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767320446 with bad export cookie 1347238798835401418 [ 5690.686748] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5690.688018] LustreError: 246727:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5690.816144] LustreError: 250870:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5690.821911] LustreError: 250870:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5690.951946] Lustre: server umount lustre-MDT0001 complete [ 5701.566737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5701.957922] LustreError: 251540:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5701.977332] LustreError: 251540:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5702.029562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5705.817235] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5707.234089] LustreError: 251539:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5707.244673] LustreError: 251539:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5712.353203] LustreError: 251539:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5716.664926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5716.960315] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5720.470789] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5725.440452] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5728.612926] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5737.540105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5737.827374] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5743.130961] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5747.744541] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5751.107467] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5751.274668] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5754.697360] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5754.894714] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5771.350175] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5771.427094] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5771.600285] Lustre: lustre-OST0001: new disk, initializing [ 5771.609478] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5771.705889] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5777.343358] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 5781.626245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5781.644778] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5781.733315] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5787.263337] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5787.407652] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5792.180647] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5792.534525] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5796.366477] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff90030860b000.ost_server_uuid 50 [ 5797.976949] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff90030860b000.ost_server_uuid in IDLE state after 0 sec [ 5807.593436] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5807.595228] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5807.602574] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5807.646612] Lustre: Skipped 1 previous similar message [ 5812.713130] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5812.715986] Lustre: Skipped 2 previous similar messages [ 5812.750934] LustreError: 256715:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5812.758178] LustreError: 256715:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5812.926530] Lustre: server umount lustre-OST0001 complete [ 5817.827247] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5817.832964] LustreError: 255942:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5817.839757] Lustre: Skipped 1 previous similar message [ 5817.865194] LustreError: 255942:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5822.337147] LustreError: 256917:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5822.346310] LustreError: 256917:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5822.591287] Lustre: server umount lustre-OST0000 complete [ 5827.728307] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5827.730222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5827.752819] Lustre: Skipped 1 previous similar message [ 5827.776200] Lustre: Skipped 3 previous similar messages [ 5842.399216] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5842.511351] LustreError: 257167:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5842.526882] LustreError: 257167:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5842.721334] Lustre: server umount lustre-MDT0000 complete [ 5843.426287] LustreError: 251540:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5843.441029] LustreError: 251540:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5847.608646] LustreError: 251519:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767320603 with bad export cookie 1347238798835402293 [ 5847.612741] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5847.625103] LustreError: 251519:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5847.892345] Lustre: server umount lustre-MDT0001 complete [ 5872.250757] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 5874.852371] Key type lgssc unregistered [ 5875.159066] LNet: 258649:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5875.167267] LNetError: 258649:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5875.179942] LNet: Removed LNI 192.168.204.105@tcp [ 5875.887725] Key type .llcrypt unregistered [ 5875.891075] Key type ._llcrypt unregistered [ 5921.195553] Key type ._llcrypt registered [ 5921.206651] Key type .llcrypt registered [ 5921.333217] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 5980.907812] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 5981.936883] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5981.960120] alg: No test for adler32 (adler32-zlib) [ 5982.966561] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [ 5983.143989] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 5984.855175] Key type lgssc registered [ 5985.822302] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6000.939223] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6000.967526] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6002.224807] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6004.481729] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6018.120404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6018.171849] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6018.181439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6019.346826] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6019.370583] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6019.428320] Lustre: lustre-MDT0000: new disk, initializing [ 6019.536420] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6019.550605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6023.643827] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6038.260204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6038.323684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6038.378201] Lustre: 268880:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6038.434512] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6038.437261] Lustre: Skipped 1 previous similar message [ 6038.518621] Lustre: lustre-MDT0001: new disk, initializing [ 6038.645852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6038.694654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6038.707696] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6042.390194] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6048.752752] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6051.681590] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6063.149293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6063.235398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6063.502353] Lustre: lustre-OST0000: new disk, initializing [ 6063.511776] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6063.610458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6069.076775] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6072.947997] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6072.956840] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6073.100248] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6076.759580] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6079.660504] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6079.851264] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6083.551809] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6083.750392] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6088.684806] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6088.685436] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6088.693837] LustreError: Skipped 1 previous similar message [ 6088.695199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6088.717660] Lustre: Skipped 1 previous similar message [ 6088.741989] Lustre: Skipped 1 previous similar message [ 6091.643045] LustreError: 272508:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6091.778252] Lustre: server umount lustre-OST0000 complete [ 6098.913592] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6098.938387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6102.566908] LustreError: 272759:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6102.569537] LustreError: 272759:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6102.641637] Lustre: server umount lustre-MDT0000 complete [ 6107.084740] LustreError: 268870:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767320863 with bad export cookie 1417326812754785097 [ 6107.091646] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6107.093611] LustreError: 268870:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6107.201454] LustreError: 273009:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6107.205332] LustreError: 273009:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6107.325453] Lustre: server umount lustre-MDT0001 complete [ 6116.131382] Lustre: DEBUG MARKER: == conf-sanity test 47a: server restart does not lose lru_resize ========================================================== 21:27:49 (1767320869) [ 6142.237922] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 6201.150752] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 6219.845633] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6220.639856] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6221.950359] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6223.829280] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6237.830390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6237.903460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6238.128651] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6238.161904] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6238.238218] Lustre: lustre-MDT0000: new disk, initializing [ 6238.312703] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6238.330406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6242.343796] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6257.994202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6258.045206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6258.134756] Lustre: 281911:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6258.222529] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6258.225868] Lustre: Skipped 1 previous similar message [ 6258.284221] Lustre: lustre-MDT0001: new disk, initializing [ 6258.365041] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6258.390372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6258.401325] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6262.463624] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6269.908164] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6273.407918] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6284.325972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6284.407316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6284.649981] Lustre: lustre-OST0000: new disk, initializing [ 6284.654368] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6284.732809] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6290.380531] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6292.032337] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6292.040396] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6292.137921] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6297.716174] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6302.035968] Lustre: Failing over lustre-OST0000 [ 6302.079993] LustreError: 285043:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6302.082594] LustreError: 285043:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 6302.202710] Lustre: server umount lustre-OST0000 complete [ 6318.051549] Lustre: 265356:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767321058/real 1767321058] req@00000000507d0d5f x1853170111288320/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1767321074 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6318.051922] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6318.078476] Lustre: 265356:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6318.095877] Lustre: Skipped 2 previous similar messages [ 6319.072136] Lustre: 265354:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767321059/real 1767321059] req@0000000068033b71 x1853170111289088/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1767321075 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6324.191565] Lustre: 265353:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767321064/real 1767321064] req@000000004d61fbec x1853170111290112/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1767321080 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6328.345071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6328.552944] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6328.565915] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 6329.311186] Lustre: 265354:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767321069/real 1767321069] req@00000000a0aecca6 x1853170111291136/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1767321085 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6329.347342] Lustre: 265354:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6331.233966] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 6333.044572] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6336.821779] Lustre: Failing over lustre-MDT0000 [ 6336.963075] LustreError: 286456:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6336.971163] LustreError: 286456:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6337.136037] Lustre: server umount lustre-MDT0000 complete [ 6339.042748] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6339.051048] LustreError: 281918:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6339.053410] Lustre: Skipped 2 previous similar messages [ 6339.063800] LustreError: 281918:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6341.515689] LustreError: 281917:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6341.550663] LustreError: 281917:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6343.205710] LustreError: 281917:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6343.219968] LustreError: 281917:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6346.593800] LustreError: 281917:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6351.719376] LustreError: 281918:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6351.739903] LustreError: 281918:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6355.423505] Lustre: 265353:0:(client.c:2478:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1767321095/real 1767321095] req@00000000bc0c9f58 x1853170111300608/t0(0) o400->MGC192.168.204.105@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1767321111 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6355.454802] Lustre: 265353:0:(client.c:2478:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6355.464905] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6360.544672] LustreError: 283500:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6360.558902] LustreError: 283500:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 9 previous similar messages [ 6366.943947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6377.320446] LustreError: 281917:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6377.335452] LustreError: 281917:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 6380.002318] LustreError: 265352:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000581e9ad1 x1853170111306624/t0(0) o250->MGC192.168.204.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6380.296104] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 6380.390316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6380.461883] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 6382.450840] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6384.115284] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6385.647628] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6385.666060] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 6385.731938] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6395.876288] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6395.877645] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6395.880316] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6395.880324] Lustre: Skipped 1 previous similar message [ 6395.882225] Lustre: Skipped 1 previous similar message [ 6395.887563] LustreError: Skipped 1 previous similar message [ 6398.426661] LustreError: 287924:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6398.431027] LustreError: 287924:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6398.564522] Lustre: server umount lustre-OST0000 complete [ 6406.112462] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6406.114481] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6406.123848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6406.123856] Lustre: Skipped 1 previous similar message [ 6406.149846] Lustre: Skipped 1 previous similar message [ 6408.848894] Lustre: server umount lustre-MDT0000 complete [ 6411.232151] LustreError: 283500:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6411.251169] LustreError: 283500:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6413.124230] LustreError: 281903:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767321169 with bad export cookie 1417326812754787204 [ 6413.128448] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6413.132896] LustreError: 281903:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6413.401363] Lustre: server umount lustre-MDT0001 complete [ 6425.971727] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 6428.575461] Key type lgssc unregistered [ 6428.917387] LNet: 289102:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6428.927128] LNetError: 289102:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6428.958061] LNet: Removed LNI 192.168.204.105@tcp [ 6429.692849] Key type .llcrypt unregistered [ 6429.694895] Key type ._llcrypt unregistered [ 6532.527356] Key type ._llcrypt registered [ 6532.530551] Key type .llcrypt registered [ 6532.632259] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 6534.314222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6534.340519] alg: No test for adler32 (adler32-zlib) [ 6535.434285] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [ 6535.633458] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 6537.367147] Key type lgssc registered [ 6538.241569] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6547.439578] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6547.460970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6548.944953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6553.259622] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6559.206300] LustreError: 294470:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6559.225273] LustreError: 294470:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6560.434202] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6564.329466] LustreError: 294470:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6564.597975] LustreError: 294499:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6564.612736] LustreError: 294499:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6564.625402] LustreError: 294499:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 0, failed: rc = -5 [ 6564.910652] LustreError: 295166:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6565.194872] Lustre: server umount lustre-MDT0000 complete [ 6592.258017] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 6663.799459] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 6686.884236] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6687.197596] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6688.978625] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6691.788922] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6706.422739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6706.461812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6706.642598] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6706.676260] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6706.743926] Lustre: lustre-MDT0000: new disk, initializing [ 6706.840645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6706.859079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6710.699835] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6724.720360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6724.786302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6724.851931] Lustre: 303634:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6724.885551] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6724.887584] Lustre: Skipped 1 previous similar message [ 6724.952635] Lustre: lustre-MDT0001: new disk, initializing [ 6725.050934] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6725.078982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6725.083528] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6728.769948] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6735.278727] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6738.803390] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6751.747648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6751.836627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6752.124219] Lustre: lustre-OST0000: new disk, initializing [ 6752.137487] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6752.242903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6758.258546] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 6759.472870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6759.487645] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6759.589214] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6766.087986] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6771.066985] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 6772.538689] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 21:38:46 (1767321526) [ 7000.543923] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7000.544519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7000.560205] Lustre: Skipped 1 previous similar message [ 7000.567142] Lustre: Skipped 1 previous similar message [ 7004.792628] LustreError: 308587:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7004.801249] LustreError: 308587:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7004.887856] Lustre: server umount lustre-MDT0001 complete [ 7005.189539] Lustre: server umount lustre-MDT0000 complete [ 7014.615486] LustreError: 303626:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767321770 with bad export cookie 15621388574890200298 [ 7014.621058] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7014.625424] LustreError: 303626:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7020.723922] LustreError: 309561:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 7020.728967] LustreError: 309561:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 7020.882928] Lustre: server umount lustre-OST0000 complete [ 7035.645907] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 7098.894505] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 7117.670928] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7118.731566] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7120.369877] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7122.983301] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7137.181541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7137.237563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7137.448691] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7137.472802] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7137.554565] Lustre: lustre-MDT0000: new disk, initializing [ 7137.668320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7137.692235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7141.348345] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 7157.350419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7157.408669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7157.446948] Lustre: 317088:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7157.504735] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7157.508984] Lustre: Skipped 1 previous similar message [ 7157.622428] Lustre: lustre-MDT0001: new disk, initializing [ 7157.697835] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7157.721412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7157.729899] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7163.055366] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 7171.729880] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7177.336916] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7195.738330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7195.826283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7196.203550] Lustre: lustre-OST0000: new disk, initializing [ 7196.210241] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7196.318701] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7204.588399] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 7205.923853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7205.934142] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7206.111025] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7213.475247] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7219.434695] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7219.704407] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7225.573282] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7225.840480] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7231.460497] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7231.476425] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7231.518239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7234.330408] LustreError: 320721:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7234.340568] LustreError: 320721:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7234.478727] Lustre: server umount lustre-OST0000 complete [ 7239.653457] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7239.662275] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7239.672749] Lustre: Skipped 1 previous similar message [ 7239.676716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7239.686443] Lustre: Skipped 1 previous similar message [ 7242.209524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7242.219637] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7245.434821] LustreError: 320970:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 7245.444309] LustreError: 320970:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7245.572260] Lustre: server umount lustre-MDT0000 complete [ 7247.332553] LustreError: 317094:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7247.345534] LustreError: 317094:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7250.683779] LustreError: 317080:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767322006 with bad export cookie 15621388574890359135 [ 7250.687853] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7250.692452] LustreError: 317080:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7251.119224] Lustre: server umount lustre-MDT0001 complete [ 7261.336442] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 21:46:54 (1767322014) [ 7293.398211] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 7361.477677] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 7382.264368] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7382.534328] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7384.277459] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7387.023043] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7402.118878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7402.244631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7402.570402] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7402.618548] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7402.701257] Lustre: lustre-MDT0000: new disk, initializing [ 7402.784977] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7402.804548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7407.400238] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 7423.409918] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7423.467330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7423.549382] Lustre: 330127:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7423.792229] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7427.924742] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 7435.689531] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7439.650145] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7455.531478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7455.600578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7455.861157] Lustre: lustre-OST0000: new disk, initializing [ 7455.864695] Lustre: Skipped 1 previous similar message [ 7455.869356] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7455.875807] Lustre: Skipped 2 previous similar messages [ 7464.069043] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 7465.623291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7465.653310] Lustre: Skipped 1 previous similar message [ 7465.665911] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7465.839415] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7476.597049] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7486.433691] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7486.448292] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7486.455583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7486.470648] Lustre: Skipped 1 previous similar message [ 7486.487096] Lustre: Skipped 2 previous similar messages [ 7492.230209] LustreError: 333365:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7492.235430] LustreError: 333365:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 7492.286167] Lustre: server umount lustre-OST0000 complete [ 7500.776110] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7500.807042] LustreError: Skipped 1 previous similar message [ 7500.820714] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7500.874369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7500.882959] Lustre: Skipped 2 previous similar messages [ 7504.247808] Lustre: server umount lustre-MDT0000 complete [ 7506.912732] LustreError: 330132:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7506.932240] LustreError: 330132:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7509.276947] LustreError: 330117:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767322265 with bad export cookie 15621388574890360017 [ 7509.278865] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7509.292727] LustreError: 330117:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7509.452528] LustreError: 333864:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7509.461612] LustreError: 333864:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 7509.759653] Lustre: server umount lustre-MDT0001 complete [ 7520.416688] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 21:51:14 (1767322274) [ 7548.352724] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 7620.168240] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 7641.620182] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7641.752687] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7644.667737] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7647.138819] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7663.017113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7663.102644] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7663.303793] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7663.332438] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7663.405485] Lustre: lustre-MDT0000: new disk, initializing [ 7663.490831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7663.496820] Lustre: Skipped 2 previous similar messages [ 7663.511048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7667.436514] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 7684.246258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7684.357645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7684.541706] Lustre: 342773:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7684.851138] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7690.364108] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 7698.726921] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7702.906569] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7716.706671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7716.789283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7723.314881] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 7727.192304] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7727.377593] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7731.733843] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7742.945589] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7742.957576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7742.958615] Lustre: Skipped 2 previous similar messages [ 7742.972367] Lustre: Skipped 3 previous similar messages [ 7744.634200] LustreError: 346011:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 7744.636732] LustreError: 346011:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 7744.736230] Lustre: server umount lustre-OST0000 complete [ 7751.648113] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7755.231857] Lustre: server umount lustre-MDT0000 complete [ 7758.307560] LustreError: 342783:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7758.321057] LustreError: 342783:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7759.231564] LustreError: 342763:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767322515 with bad export cookie 15621388574890361102 [ 7759.234996] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7759.242023] LustreError: 342763:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7759.581321] Lustre: server umount lustre-MDT0001 complete [ 7773.843648] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 7776.306816] Key type lgssc unregistered [ 7776.526345] LNet: 347189:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7776.530081] LNetError: 347189:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7776.543472] LNet: Removed LNI 192.168.204.105@tcp [ 7777.194511] Key type .llcrypt unregistered [ 7777.196342] Key type ._llcrypt unregistered [ 7858.220191] Key type ._llcrypt registered [ 7858.222936] Key type .llcrypt registered [ 7858.392465] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 7859.840891] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7859.866249] alg: No test for adler32 (adler32-zlib) [ 7860.934563] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [ 7861.128003] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 7862.799847] Key type lgssc registered [ 7863.887741] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7871.659213] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7871.677372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7873.089248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7877.521679] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 7883.232349] LustreError: 351645:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7883.253065] LustreError: 351645:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7888.354466] LustreError: 351645:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7890.388100] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7890.933504] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7894.819859] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 7900.492252] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7904.096637] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7912.563341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 7912.850622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7918.019757] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 7918.734490] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 7924.311043] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7943.651164] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7943.668301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7943.671177] Lustre: Skipped 1 previous similar message [ 7948.487307] LustreError: 354299:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 7948.735501] Lustre: server umount lustre-OST0000 complete [ 7953.687929] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7953.692745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7953.710018] Lustre: Skipped 1 previous similar message [ 7953.716417] Lustre: Skipped 1 previous similar message [ 7959.009907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7959.016475] Lustre: Skipped 2 previous similar messages [ 7959.919761] LustreError: 354549:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 7959.924875] LustreError: 354549:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 7960.036827] Lustre: server umount lustre-MDT0000 complete [ 7963.110694] LustreError: 351646:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7963.143639] LustreError: 351646:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7964.697299] LustreError: 351627:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767322720 with bad export cookie 8522287579375577753 [ 7964.704111] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7964.708250] LustreError: 351627:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7964.802220] LustreError: 354800:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 7964.807899] LustreError: 354800:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 7965.001601] Lustre: server umount lustre-MDT0001 complete [ 7977.844244] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 7980.143530] Key type lgssc unregistered [ 7980.433519] LNet: 355479:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7980.438666] LNetError: 355479:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7980.454573] LNet: Removed LNI 192.168.204.105@tcp [ 7981.181843] Key type .llcrypt unregistered [ 7981.184111] Key type ._llcrypt unregistered [ 8052.344417] Key type ._llcrypt registered [ 8052.347171] Key type .llcrypt registered [ 8052.443068] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 8053.197710] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8053.339634] alg: No test for adler32 (adler32-zlib) [ 8054.366895] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [ 8054.542838] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 8056.191517] Key type lgssc registered [ 8057.464992] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8063.914672] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8063.948704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8065.361538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8069.857391] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 8075.745629] LustreError: 359936:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8075.776949] LustreError: 359936:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8080.311229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8080.607453] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8083.884512] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 8088.351442] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8091.635433] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8100.329881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8100.593752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8106.224332] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 8110.840499] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 8111.381936] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8119.160739] Lustre: lustre-OST0000: Not available for connect from 192.168.204.5@tcp (stopping) [ 8121.313815] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8121.326716] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8121.328428] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8123.462134] LustreError: 362576:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8123.638138] Lustre: server umount lustre-OST0000 complete [ 8128.865980] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff900310fcd000.ost_server_uuid 50 [ 8130.381217] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 8146.914574] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8146.923126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8146.936469] Lustre: Skipped 2 previous similar messages [ 8146.940752] Lustre: Skipped 1 previous similar message [ 8150.300466] LustreError: 362992:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 8150.304822] LustreError: 362992:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8150.372443] Lustre: server umount lustre-MDT0000 complete [ 8152.032259] LustreError: 359937:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8152.044597] LustreError: 359937:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8153.695443] LustreError: 359917:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767322909 with bad export cookie 9960952974999272437 [ 8153.708499] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8153.867508] LustreError: 363241:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 8153.877914] LustreError: 363241:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 8154.135255] Lustre: server umount lustre-MDT0001 complete [ 8161.027472] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 22:01:55 (1767322915) [ 8169.721523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8170.031462] LustreError: 364106:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8170.093166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8173.633812] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 8175.595367] LustreError: 364106:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8180.709858] LustreError: 364106:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8182.942800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8183.227716] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8186.886816] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 8191.744527] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8194.786934] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8202.612341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8202.888635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8207.903627] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 8212.602666] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8215.230495] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8226.826548] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8226.881593] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8227.011607] Lustre: lustre-OST0001: new disk, initializing [ 8227.015488] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8227.097856] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8233.428637] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 8236.087027] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8236.096038] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8236.173861] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8240.879855] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8244.194910] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8244.198539] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8244.210694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8244.216658] Lustre: Skipped 1 previous similar message [ 8246.241149] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8246.250920] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8249.154381] LustreError: 368233:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8249.159790] LustreError: 368233:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 8249.316931] Lustre: server umount lustre-OST0000 complete [ 8251.749341] LustreError: 366085:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8254.873779] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8255.048384] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 8259.083203] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8259.340456] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 8270.309697] LustreError: 366087:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8270.320121] LustreError: 366087:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 8272.356907] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8272.364643] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8272.381956] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8272.386385] Lustre: Skipped 3 previous similar messages [ 8278.201918] LustreError: 368956:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 8278.206599] LustreError: 368956:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8278.295442] Lustre: server umount lustre-OST0001 complete [ 8285.665348] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8285.666055] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8285.677179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8285.677186] Lustre: Skipped 2 previous similar messages [ 8285.697901] Lustre: Skipped 2 previous similar messages [ 8288.131562] LustreError: 369206:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 8288.133702] LustreError: 369206:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 8288.271663] Lustre: server umount lustre-MDT0000 complete [ 8292.147362] LustreError: 364086:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767323048 with bad export cookie 9960952974999273508 [ 8292.152649] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8292.161495] LustreError: 364086:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8292.499181] Lustre: server umount lustre-MDT0001 complete [ 8298.114786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8302.218883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8306.731503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8311.074397] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8316.856176] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 22:04:31 (1767323071) [ 8325.184790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8325.400763] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8325.423509] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8325.558539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8329.656439] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 8340.864548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8340.943678] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8340.976919] Lustre: 371337:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8344.679292] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 8350.054666] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8353.512827] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8362.789694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8362.961268] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8369.028874] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 8370.938567] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 8376.081708] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8391.895552] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8392.037659] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 8392.357183] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8392.372547] Lustre: Skipped 2 previous similar messages [ 8398.572913] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 8399.520717] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 8405.556324] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8410.088827] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8410.118471] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8410.136277] Lustre: Skipped 2 previous similar messages [ 8423.391692] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8423.463810] LustreError: 375305:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8423.471455] LustreError: 375305:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 8423.521348] Lustre: server umount lustre-OST0000 complete [ 8425.442517] LustreError: 373352:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8425.456706] LustreError: 373352:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 8440.302666] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8443.210709] Lustre: server umount lustre-OST0001 complete [ 8448.992476] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8449.006711] Lustre: Skipped 3 previous similar messages [ 8449.010099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8449.016993] Lustre: Skipped 9 previous similar messages [ 8453.413682] Lustre: server umount lustre-MDT0000 complete [ 8457.459296] LustreError: 372262:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767323213 with bad export cookie 9960952974999274915 [ 8457.461604] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8457.470336] LustreError: 372262:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8457.595600] LustreError: 376028:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 8457.606043] LustreError: 376028:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [ 8457.740444] Lustre: server umount lustre-MDT0001 complete [ 8464.319779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8468.469738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8472.555896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8476.832482] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8482.435943] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 22:07:16 (1767323236) [ 8508.702351] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 8560.485836] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 8577.270609] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8577.430332] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8578.800420] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8580.446274] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8592.827710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8592.894459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8593.105499] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8593.125823] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8593.190296] Lustre: lustre-MDT0000: new disk, initializing [ 8593.230115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8593.239892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8596.461778] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 8609.761616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8609.829830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8609.938383] Lustre: 386154:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8610.001261] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8610.004798] Lustre: Skipped 1 previous similar message [ 8610.129103] Lustre: lustre-MDT0001: new disk, initializing [ 8610.256889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8610.264625] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8613.416246] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 8620.029933] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8623.390474] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8635.138372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8635.198799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8635.410880] Lustre: lustre-OST0000: new disk, initializing [ 8635.415341] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8640.518602] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 8641.600901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8641.605966] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8641.697163] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8647.607298] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8652.090726] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8652.301539] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8655.886198] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8656.079350] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8662.499835] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8662.505775] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8662.507983] LustreError: Skipped 3 previous similar messages [ 8662.513645] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8662.529580] Lustre: Skipped 2 previous similar messages [ 8662.548829] Lustre: Skipped 3 previous similar messages [ 8663.752203] LustreError: 389784:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 8663.755890] LustreError: 389784:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 8663.911927] Lustre: server umount lustre-OST0000 complete [ 8675.343814] Lustre: server umount lustre-MDT0000 complete [ 8677.859518] LustreError: 386160:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8677.879096] LustreError: 386160:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 8681.080285] LustreError: 386147:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767323437 with bad export cookie 9960952974999276707 [ 8681.099466] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8681.520618] Lustre: server umount lustre-MDT0001 complete [ 8693.368813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8698.131438] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 8709.000475] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8713.282147] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 8718.149627] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8720.886807] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8729.519295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8734.767665] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 8739.515964] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8742.996955] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8743.259568] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8746.508282] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8746.672850] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8749.538092] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8749.553219] LustreError: Skipped 1 previous similar message [ 8762.335255] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8762.548501] Lustre: server umount lustre-OST0000 complete [ 8768.218604] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8772.009493] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8802.205215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8807.976828] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 8812.514748] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8815.802418] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8815.820504] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8815.822624] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8815.839826] Lustre: Skipped 1 previous similar message [ 8819.020297] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 5 sec [ 8826.336790] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8826.339425] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8826.340213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8826.340218] Lustre: Skipped 9 previous similar messages [ 8826.340980] LustreError: Skipped 1 previous similar message [ 8826.352812] Lustre: Skipped 5 previous similar messages [ 8836.064498] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8836.101069] LustreError: 395916:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 8836.109711] LustreError: 395916:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 8836.244417] Lustre: server umount lustre-OST0000 complete [ 8846.155559] Lustre: server umount lustre-MDT0000 complete [ 8846.820089] LustreError: 390954:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8846.843352] LustreError: 390954:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 8849.724296] LustreError: 390934:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767323605 with bad export cookie 9960952974999277589 [ 8849.725340] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8849.730526] LustreError: 390934:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8850.123937] Lustre: server umount lustre-MDT0001 complete [ 8856.356183] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 22:13:30 (1767323610) [ 8863.735135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8864.068813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8864.075502] Lustre: Skipped 6 previous similar messages [ 8866.914317] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 8877.142284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8881.451849] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 8886.421687] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8889.544320] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8897.812701] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8903.318457] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 8908.237242] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8911.802306] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8911.989472] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8915.246852] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8915.423104] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8928.399857] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 8928.495983] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8928.580109] Lustre: lustre-OST0001: new disk, initializing [ 8928.586160] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8933.298890] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 8934.471874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8934.478277] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8934.539489] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8939.901746] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8942.999660] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 8943.148679] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8946.488395] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 8946.610267] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8953.954606] Lustre: server umount lustre-OST0001 complete [ 8957.892038] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 8958.061579] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 8961.277498] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 8961.468860] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 8991.705554] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 8996.359285] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 8997.353839] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 8997.368449] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 8997.369705] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9000.001753] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9001.208513] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9002.470576] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9002.473895] LustreError: Skipped 1 previous similar message [ 9008.576689] Lustre: server umount lustre-OST0001 complete [ 9028.439448] Lustre: server umount lustre-OST0000 complete [ 9037.187493] Lustre: server umount lustre-MDT0000 complete [ 9039.801016] LustreError: 397258:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767323796 with bad export cookie 9960952974999278555 [ 9039.805424] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9039.806680] LustreError: 397258:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9054.175576] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9054.337116] Lustre: server umount lustre-MDT0001 complete [ 9058.515245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9061.816722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9064.770207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9067.928553] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9072.158681] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 22:17:06 (1767323826) [ 9079.159027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9079.378828] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9079.394128] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9082.111987] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 9090.780629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9090.853587] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9090.871977] Lustre: 406856:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9093.527444] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 9097.139365] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9099.816362] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9106.490508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9106.610529] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9110.919436] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 9115.315669] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9125.218878] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9125.266902] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9128.807622] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 9133.031333] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9134.012910] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9136.612159] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 9136.756607] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9139.288392] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 9139.434866] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9141.903154] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff900318eb1000.ost_server_uuid 50 [ 9143.206123] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff900318eb1000.ost_server_uuid in FULL state after 0 sec [ 9143.882416] Lustre: Permanently deactivating lustre-OST0001 [ 9143.890110] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 9145.966488] Lustre: Permanently reactivating lustre-OST0001 [ 9151.968819] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9151.977422] Lustre: Skipped 10 previous similar messages [ 9151.981398] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9151.984735] Lustre: Skipped 20 previous similar messages [ 9154.621113] LustreError: 411607:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 9154.625019] LustreError: 411607:0:(obd_class.h:479:obd_check_dev()) Skipped 47 previous similar messages [ 9154.698080] Lustre: server umount lustre-OST0001 complete [ 9157.606867] LustreError: 408860:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9157.618066] LustreError: 408860:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 36 previous similar messages [ 9163.514933] Lustre: server umount lustre-OST0000 complete [ 9172.673647] Lustre: server umount lustre-MDT0000 complete [ 9175.695906] LustreError: 406846:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767323931 with bad export cookie 9960952974999279843 [ 9175.697628] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9175.702187] LustreError: 406846:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9175.941084] Lustre: server umount lustre-MDT0001 complete [ 9180.160318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9183.563559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9186.714566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9190.255082] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9195.015047] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 22:19:09 (1767323949) [ 9196.228635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9202.773323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9202.939828] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9202.957955] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9202.963796] Lustre: Skipped 5 previous similar messages [ 9205.693959] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 9213.497771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9213.574207] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9213.601230] Lustre: 414296:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9216.630461] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 9220.524708] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9223.401680] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9231.587132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9231.658902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9231.820605] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9231.837904] Lustre: Permanently deactivating lustre-OST0000 [ 9231.849478] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 9236.621880] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 9239.056604] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9240.297331] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9250.119596] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9250.180827] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9254.815890] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 9255.624741] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 9260.220285] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9261.621854] Lustre: Permanently reactivating lustre-OST0000 [ 9261.642976] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 9261.648556] Lustre: Skipped 2 previous similar messages [ 9262.335576] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 9262.349349] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 9262.358244] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9262.366990] Lustre: Skipped 1 previous similar message [ 9262.407109] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 9262.428897] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 9276.392849] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9276.397291] LustreError: Skipped 5 previous similar messages [ 9277.424520] Lustre: server umount lustre-OST0001 complete [ 9294.303131] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9294.387816] Lustre: server umount lustre-OST0000 complete [ 9303.478545] Lustre: server umount lustre-MDT0000 complete [ 9306.845746] LustreError: 414288:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767324063 with bad export cookie 9960952974999281488 [ 9306.852612] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9306.856459] LustreError: 414288:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9307.190771] Lustre: server umount lustre-MDT0001 complete [ 9316.151616] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 9317.693758] Key type lgssc unregistered [ 9317.882391] LNet: 419727:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9317.890720] LNetError: 419727:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9317.905123] LNet: Removed LNI 192.168.204.105@tcp [ 9318.421576] Key type .llcrypt unregistered [ 9318.422946] Key type ._llcrypt unregistered [ 9366.048698] Key type ._llcrypt registered [ 9366.050589] Key type .llcrypt registered [ 9366.146180] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 9366.712878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9366.789619] alg: No test for adler32 (adler32-zlib) [ 9367.718307] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [ 9367.876590] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 9369.504934] Key type lgssc registered [ 9370.010055] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9372.409864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9378.082418] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9378.104609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9379.405812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9381.763618] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 9388.686725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9388.734361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9388.776980] Lustre: Found index 1 for lustre-MDT0001, updating log [ 9388.795426] Lustre: 424272:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9388.802683] Lustre: Permanently deactivating lustre-MDT0001 [ 9388.811056] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 9388.846258] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 9388.850814] LustreError: 424313:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 9388.862303] LustreError: 424313:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 9388.875960] LustreError: 424313:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 1, failed: rc = -108 [ 9389.018889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9391.495621] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 9394.788674] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9397.252887] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9404.271803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9404.526373] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9405.540413] LustreError: 426286:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9405.550702] LustreError: 426286:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9405.556938] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 9407.847834] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 9411.043382] LustreError: 426286:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9411.284761] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9414.116637] LustreError: 426286:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9414.129704] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 9414.135461] LustreError: 426286:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9419.236144] LustreError: 426401:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9419.247310] LustreError: 426401:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9419.945762] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9420.072600] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9423.884523] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 9425.390517] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 9425.392952] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 9428.515623] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9432.015304] Lustre: Permanently reactivating lustre-MDT0001 [ 9432.023867] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 9432.027155] Lustre: Skipped 2 previous similar messages [ 9434.176282] Lustre: lustre-MDT0001: Client bc363f9f-e90d-46ef-96a1-0feb97cfaae0 (at 192.168.204.5@tcp) reconnecting [ 9436.399310] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9437.375550] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9440.455439] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 9440.572534] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 9442.664524] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 9442.781053] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 9444.296790] Lustre: Permanently deactivating lustre-MDT0001 [ 9444.304934] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 9444.307991] Lustre: Skipped 2 previous similar messages [ 9450.521846] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 9455.585717] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9455.595612] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9455.604729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9459.168888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9459.170280] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9459.171658] Lustre: Skipped 1 previous similar message [ 9459.195094] Lustre: Skipped 2 previous similar messages [ 9461.324168] LustreError: 429167:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 9461.386564] Lustre: server umount lustre-MDT0000 complete [ 9464.291380] LustreError: 424283:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9464.302239] LustreError: 424283:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9464.347923] LustreError: 428158:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767324220 with bad export cookie 18390716781503952113 [ 9464.351179] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9464.352612] LustreError: 428158:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9464.442173] LustreError: 429416:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 9464.447198] LustreError: 429416:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 9464.523616] Lustre: server umount lustre-MDT0001 complete [ 9473.465131] LustreError: 429616:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 9473.467468] LustreError: 429616:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 9473.491691] Lustre: server umount lustre-OST0000 complete [ 9480.108657] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 22:23:54 (1767324234) [ 9487.464340] LustreError: 431195:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 9487.467158] LustreError: 431195:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 9487.613156] Lustre: server umount lustre-OST0001 complete [ 9498.712868] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 9539.172268] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 9550.432936] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9550.520363] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9551.604707] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9553.033953] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9563.000500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9563.039442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9563.174720] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9563.178513] Lustre: Skipped 2 previous similar messages [ 9563.198835] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9563.247575] Lustre: lustre-MDT0000: new disk, initializing [ 9563.313945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9563.327932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9566.063716] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 9576.477286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9576.531481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9576.591921] Lustre: 438913:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9576.639669] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9576.642570] Lustre: Skipped 1 previous similar message [ 9576.717655] Lustre: lustre-MDT0001: new disk, initializing [ 9576.782835] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9576.809976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9576.821882] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9579.426659] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 9584.799960] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9587.396846] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9595.959994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9596.014451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9596.208222] Lustre: lustre-OST0000: new disk, initializing [ 9596.214412] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9596.268244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9600.507832] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 9601.575604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9601.582466] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9601.626077] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9605.819514] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9607.241620] LustreError: 440226:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 9617.272037] LustreError: 440226:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 9617.287226] LustreError: 440226:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 9618.192695] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9618.227813] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9618.322856] Lustre: lustre-OST0001: new disk, initializing [ 9618.325356] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9621.555143] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 9626.142274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9626.152494] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9626.220329] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9626.874428] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9627.328073] LustreError: 440226:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 9627.834707] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9631.200332] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9631.210447] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9631.218410] Lustre: Skipped 1 previous similar message [ 9631.219154] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9631.224538] Lustre: Skipped 3 previous similar messages [ 9634.929476] LustreError: 443634:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 9634.933017] LustreError: 443634:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9635.006219] Lustre: server umount lustre-OST0001 complete [ 9636.325406] LustreError: 441334:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9636.347208] LustreError: 441334:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9651.683111] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9651.685768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9651.697714] Lustre: Skipped 1 previous similar message [ 9651.700922] Lustre: Skipped 1 previous similar message [ 9654.836146] LustreError: 443860:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9654.845424] LustreError: 443860:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9654.895290] Lustre: server umount lustre-OST0000 complete [ 9658.850245] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9658.854277] LustreError: Skipped 1 previous similar message [ 9658.856151] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9658.863070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9658.867902] Lustre: Skipped 1 previous similar message [ 9663.745788] Lustre: server umount lustre-MDT0000 complete [ 9666.211687] LustreError: 438904:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767324422 with bad export cookie 18390716781503954927 [ 9666.212867] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9666.218400] LustreError: 438904:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9672.430344] LustreError: 444358:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 9672.434326] LustreError: 444358:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 9672.539859] Lustre: server umount lustre-MDT0001 complete [ 9678.414915] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 9679.736527] Key type lgssc unregistered [ 9679.929499] LNet: 445034:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9679.932441] LNetError: 445034:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9679.947969] LNet: Removed LNI 192.168.204.105@tcp [ 9680.360445] Key type .llcrypt unregistered [ 9680.361881] Key type ._llcrypt unregistered [ 9703.806215] Key type ._llcrypt registered [ 9703.807481] Key type .llcrypt registered [ 9703.864899] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [ 9737.084034] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [ 9737.671594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9737.680588] alg: No test for adler32 (adler32-zlib) [ 9738.616746] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [ 9738.765128] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [ 9740.383232] Key type lgssc registered [ 9741.007821] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9749.682284] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9749.956075] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9750.967146] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9752.215027] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9759.484913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9759.507682] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9759.514115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9760.631394] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9760.649041] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9760.687475] Lustre: lustre-MDT0000: new disk, initializing [ 9760.717551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9760.725667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9762.354972] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 9771.267669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9771.311911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9771.369549] Lustre: 455253:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9771.417456] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9771.420950] Lustre: Skipped 1 previous similar message [ 9771.471358] Lustre: lustre-MDT0001: new disk, initializing [ 9771.518103] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9771.543186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9771.548579] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9773.488328] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 9777.904511] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9779.925402] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9787.010992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 9787.060078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9787.197297] Lustre: lustre-OST0000: new disk, initializing [ 9787.201324] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9787.252509] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9790.118118] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 9794.186330] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9796.322404] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9797.145259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9797.149635] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9797.192616] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9797.447086] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 9799.092572] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9799.186666] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9802.209438] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9802.210745] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9802.213795] LustreError: Skipped 1 previous similar message [ 9802.214693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9802.220415] Lustre: Skipped 1 previous similar message [ 9806.063471] LustreError: 458887:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 9806.111122] Lustre: server umount lustre-OST0000 complete [ 9812.448726] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9812.458649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9812.462469] Lustre: Skipped 2 previous similar messages [ 9814.343066] LustreError: 459137:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 9814.345581] LustreError: 459137:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9814.395242] Lustre: server umount lustre-MDT0000 complete [ 9816.454173] LustreError: 456379:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767324572 with bad export cookie 2645782835695519250 [ 9816.456408] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9816.459074] LustreError: 456379:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9816.531631] LustreError: 459386:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9816.534223] LustreError: 459386:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 9816.613989] Lustre: server umount lustre-MDT0001 complete [ 9820.268961] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 22:29:35 (1767324575) [ 9824.532475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9824.716286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9826.399559] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 9829.857169] LustreError: 460248:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9829.864863] LustreError: 460248:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9831.584020] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9831.738112] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9833.396950] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 9835.865944] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9837.424924] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9842.336919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9842.494592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9844.891920] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 9847.206265] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9895.905639] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9895.908666] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9895.913934] Lustre: Skipped 1 previous similar message [ 9895.916562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9896.930845] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9901.872096] LustreError: 462957:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9901.874939] LustreError: 462957:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 9901.930062] Lustre: server umount lustre-OST0000 complete [ 9903.791581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 9910.519614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 9910.688795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9913.214749] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [ 9915.542409] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9916.836479] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 9916.842190] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9921.957486] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 9921.962307] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 9952.738069] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9952.740541] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9952.741528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9952.741533] Lustre: Skipped 2 previous similar messages [ 9952.744269] Lustre: Skipped 2 previous similar messages [ 9952.747404] LustreError: Skipped 1 previous similar message [ 9955.633361] LustreError: 464813:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 9955.636295] LustreError: 464813:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9955.705366] Lustre: server umount lustre-OST0000 complete [ 9959.903801] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9959.906619] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9959.912376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9959.914900] Lustre: Skipped 1 previous similar message [ 9964.105134] LustreError: 465064:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 9964.107996] LustreError: 465064:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 9964.158042] Lustre: server umount lustre-MDT0000 complete [ 9966.136212] LustreError: 460229:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767324722 with bad export cookie 2645782835695520132 [ 9966.139703] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9966.142298] LustreError: 460229:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9966.290267] Lustre: server umount lustre-MDT0001 complete [ 9972.434219] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [ 9973.707564] Key type lgssc unregistered [ 9973.861506] LNet: 465988:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9973.868458] LNetError: 465988:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9973.880650] LNet: Removed LNI 192.168.204.105@tcp [ 9974.254873] Key type .llcrypt unregistered [ 9974.256076] Key type ._llcrypt unregistered [10009.887769] Key type ._llcrypt registered [10009.889599] Key type .llcrypt registered [10009.944943] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [10010.377139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10010.421838] alg: No test for adler32 (adler32-zlib) [10011.311524] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [10011.407102] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [10012.999155] Key type lgssc registered [10013.422724] Lustre: Echo OBD driver; http://www.lustre.org/ [10016.621125] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10016.630755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10017.833121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10019.414167] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [10024.930960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10025.109870] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10026.982701] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [10029.386959] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10030.853469] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10035.530728] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10035.684303] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10038.554763] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [10041.566226] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10042.857480] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [10065.377801] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10065.386829] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10065.394394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10065.396335] Lustre: Skipped 1 previous similar message [10070.500583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10070.509784] Lustre: Skipped 1 previous similar message [10070.702802] LustreError: 473862:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10070.802979] Lustre: server umount lustre-OST0000 complete [10075.616676] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10075.628846] Lustre: Skipped 1 previous similar message [10075.634087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10079.693116] LustreError: 474112:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10079.698691] LustreError: 474112:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10080.736218] LustreError: 472935:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10080.742794] LustreError: 472935:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10081.770436] LustreError: 474112:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [10081.772921] LustreError: 474112:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10081.806290] Lustre: server umount lustre-MDT0000 complete [10084.328089] LustreError: 470418:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767324840 with bad export cookie 1546522888765364562 [10084.332479] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10084.336190] LustreError: 470418:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10084.419306] LustreError: 474361:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [10084.421492] LustreError: 474361:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [10084.528782] Lustre: server umount lustre-MDT0001 complete [10090.819371] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [10092.271353] Key type lgssc unregistered [10092.438551] LNet: 475037:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10092.442750] LNetError: 475037:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10093.485492] LNet: Removed LNI 192.168.204.105@tcp [10093.889760] Key type .llcrypt unregistered [10093.891045] Key type ._llcrypt unregistered [10127.147080] Key type ._llcrypt registered [10127.148430] Key type .llcrypt registered [10127.189622] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [10127.528713] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10127.557831] alg: No test for adler32 (adler32-zlib) [10128.439656] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [10128.537765] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [10130.143194] Key type lgssc registered [10130.586451] Lustre: Echo OBD driver; http://www.lustre.org/ [10133.880789] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10133.890526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10135.137426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10136.818479] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [10142.908056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10143.077389] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10145.168386] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [10147.925476] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10149.737438] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10154.510373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10154.654198] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10157.104172] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [10159.661846] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10160.815970] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10176.483702] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10176.493403] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10176.501290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10178.530920] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10178.530937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10178.531724] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10183.650181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10183.655233] Lustre: Skipped 2 previous similar messages [10188.780866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10188.785408] Lustre: Skipped 1 previous similar message [10189.280253] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10189.336511] LustreError: 482087:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10189.428922] Lustre: server umount lustre-OST0000 complete [10193.889337] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10193.895356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10193.898455] Lustre: Skipped 1 previous similar message [10198.157459] LustreError: 482337:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10198.160026] LustreError: 482337:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10198.194664] Lustre: server umount lustre-MDT0000 complete [10199.008497] LustreError: 479292:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10199.019284] LustreError: 479292:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10200.788573] LustreError: 482340:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767324956 with bad export cookie 12438582502834255802 [10200.789376] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10200.793977] LustreError: 482340:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10200.876150] LustreError: 482590:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [10200.882506] LustreError: 482590:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10200.983804] Lustre: server umount lustre-MDT0001 complete [10207.630219] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [10208.864353] Key type lgssc unregistered [10209.021955] LNet: 483265:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10209.025738] LNetError: 483265:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10209.038542] LNet: Removed LNI 192.168.204.105@tcp [10209.384386] Key type .llcrypt unregistered [10209.385698] Key type ._llcrypt unregistered [10250.359043] Key type ._llcrypt registered [10250.360847] Key type .llcrypt registered [10250.415483] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [10250.775154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10250.786506] alg: No test for adler32 (adler32-zlib) [10251.702289] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [10251.810336] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [10253.416211] Key type lgssc registered [10253.958572] Lustre: Echo OBD driver; http://www.lustre.org/ [10257.605886] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10257.619517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10258.863497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10261.060762] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [10267.555409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10267.817166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10269.980341] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [10273.155537] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10274.988761] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10279.864676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10280.080718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10283.191749] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [10284.202977] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [10285.831788] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10304.992933] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10305.000916] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10305.011762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10305.458157] LustreError: 491122:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10305.510108] Lustre: server umount lustre-OST0000 complete [10307.781042] LustreError: 491370:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10307.784265] LustreError: 491370:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10307.822224] Lustre: server umount lustre-MDT0000 complete [10308.066091] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10308.072872] Lustre: Skipped 1 previous similar message [10308.073389] LustreError: 489121:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10308.087784] LustreError: 489121:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10309.940024] LustreError: 487698:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767325066 with bad export cookie 16969106580469338734 [10309.945398] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10309.946156] LustreError: 487698:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10320.357541] LustreError: 491619:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [10320.362356] LustreError: 491619:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10320.437022] Lustre: server umount lustre-MDT0001 complete [10325.736144] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [10327.001481] Key type lgssc unregistered [10327.154781] LNet: 492295:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10327.158594] LNetError: 492295:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10327.168308] LNet: Removed LNI 192.168.204.105@tcp [10327.527503] Key type .llcrypt unregistered [10327.529202] Key type ._llcrypt unregistered [10358.446528] Key type ._llcrypt registered [10358.447855] Key type .llcrypt registered [10358.490582] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [10358.843763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10358.877100] alg: No test for adler32 (adler32-zlib) [10359.751064] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [10359.839195] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [10361.423171] Key type lgssc registered [10361.872897] Lustre: Echo OBD driver; http://www.lustre.org/ [10364.754694] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10364.760818] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10365.953415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10367.409069] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [10372.024801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10372.191363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10373.705296] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [10375.864597] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10377.374883] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10381.498482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10381.647360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10383.802419] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [10385.832214] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10390.886315] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [10396.129900] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10396.130555] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10396.134446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10396.139274] Lustre: Skipped 1 previous similar message [10401.006189] LustreError: 499365:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10401.044902] Lustre: server umount lustre-OST0000 complete [10402.784948] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10402.788049] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10402.795445] Lustre: Skipped 1 previous similar message [10402.801290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10406.368226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10406.368765] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10411.489851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10411.494756] Lustre: Skipped 2 previous similar messages [10416.611853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10416.616339] Lustre: Skipped 1 previous similar message [10417.119200] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10417.162660] LustreError: 499614:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10417.166479] LustreError: 499614:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10417.202423] Lustre: server umount lustre-MDT0000 complete [10419.372653] LustreError: 496531:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767325175 with bad export cookie 5513723459294700368 [10419.376148] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10419.377127] LustreError: 496531:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10419.456393] LustreError: 499864:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [10419.459978] LustreError: 499864:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10419.538578] Lustre: server umount lustre-MDT0001 complete [10425.830348] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [10427.194484] Key type lgssc unregistered [10427.342726] LNet: 500540:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10427.346769] LNetError: 500540:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10427.355663] LNet: Removed LNI 192.168.204.105@tcp [10427.756320] Key type .llcrypt unregistered [10427.758106] Key type ._llcrypt unregistered [10432.882748] Key type ._llcrypt registered [10432.884305] Key type .llcrypt registered [10432.926215] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [10444.317700] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [10474.716363] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [10475.257822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10475.265026] alg: No test for adler32 (adler32-zlib) [10476.148392] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [10476.251394] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [10477.863177] Key type lgssc registered [10478.353818] Lustre: Echo OBD driver; http://www.lustre.org/ [10485.966182] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10485.986043] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10486.828114] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10487.763336] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10494.631245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10494.654515] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10494.661061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10495.757704] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10495.773691] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10495.813454] Lustre: lustre-MDT0000: new disk, initializing [10495.844253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10495.856190] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10497.477994] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [10505.850575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10505.882112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10505.923827] Lustre: 509975:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10505.960140] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10505.963146] Lustre: Skipped 1 previous similar message [10506.003115] Lustre: lustre-MDT0001: new disk, initializing [10506.044213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10506.062876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10506.068667] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10507.756347] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [10511.514601] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10513.031613] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10518.088284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10518.117397] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10518.232531] Lustre: lustre-OST0000: new disk, initializing [10518.234700] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10518.281738] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10520.523200] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [10520.592805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10520.597383] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10520.643582] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10524.393835] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10526.267296] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10526.371375] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10528.003135] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10528.103902] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10530.787203] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10530.790524] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10530.791235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10530.792098] Lustre: Skipped 1 previous similar message [10530.801896] LustreError: Skipped 1 previous similar message [10535.035204] LustreError: 513605:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [10535.107727] Lustre: server umount lustre-OST0000 complete [10541.025339] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10541.032079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10541.037932] Lustre: Skipped 2 previous similar messages [10543.429331] LustreError: 513854:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [10543.431226] LustreError: 513854:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10543.463865] Lustre: server umount lustre-MDT0000 complete [10545.360413] LustreError: 509966:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767325301 with bad export cookie 15073792309958276307 [10545.363466] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10545.363944] LustreError: 509966:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10545.443024] LustreError: 514106:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10545.445014] LustreError: 514106:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10545.508262] Lustre: server umount lustre-MDT0001 complete [10548.678942] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 22:41:43 (1767325303) [10552.648462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10552.824364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10554.225333] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [10557.920408] LustreError: 514968:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10557.927151] LustreError: 514968:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10558.735461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10558.878530] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10560.510690] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [10562.609473] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10563.999307] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10568.008555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10568.166920] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10570.504770] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [10572.655545] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10588.642284] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10588.643626] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10588.643805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10588.646414] Lustre: Skipped 2 previous similar messages [10592.557069] LustreError: 517622:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [10592.559349] LustreError: 517622:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [10592.609709] Lustre: server umount lustre-OST0000 complete [10594.784342] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10594.787545] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10594.793469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10594.796086] Lustre: Skipped 1 previous similar message [10598.880408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10598.880791] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10600.829792] LustreError: 517871:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [10600.831572] LustreError: 517871:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10600.861793] Lustre: server umount lustre-MDT0000 complete [10602.563066] LustreError: 514948:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767325358 with bad export cookie 15073792309958277189 [10602.564994] LustreError: 515879:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [10602.566321] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10602.567809] LustreError: 514948:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10602.693092] Lustre: server umount lustre-MDT0001 complete [10608.115685] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [10609.331527] Key type lgssc unregistered [10609.467366] LNet: 518798:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10609.469548] LNetError: 518798:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10609.479172] LNet: Removed LNI 192.168.204.105@tcp [10609.778813] Key type .llcrypt unregistered [10609.780031] Key type ._llcrypt unregistered [10625.861266] Key type ._llcrypt registered [10625.862502] Key type .llcrypt registered [10625.909500] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [10656.308804] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [10656.674983] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10656.737761] alg: No test for adler32 (adler32-zlib) [10657.614341] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [10657.700200] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [10659.287150] Key type lgssc registered [10659.672837] Lustre: Echo OBD driver; http://www.lustre.org/ [10666.274107] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10666.480463] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10667.132389] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10668.090089] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10670.990262] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10673.720830] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10680.445966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10680.469792] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10680.476653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10681.582028] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10681.603712] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10681.646650] Lustre: lustre-MDT0000: new disk, initializing [10681.686605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10681.697210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10683.738810] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [10692.342120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10692.373495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10692.412924] Lustre: 528674:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10692.442189] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10692.444531] Lustre: Skipped 1 previous similar message [10692.482180] Lustre: lustre-MDT0001: new disk, initializing [10692.524597] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10692.543362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10692.547129] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10694.188614] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [10697.945813] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10699.494123] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10705.821457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10705.859740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10705.991396] Lustre: lustre-OST2710: new disk, initializing [10705.994705] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [10706.036854] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [10708.940370] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [10712.961560] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [10715.149903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [10715.157254] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [10715.210557] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [10720.164800] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10720.198020] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10720.270850] Lustre: lustre-OST03e8: new disk, initializing [10720.273207] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [10720.307448] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [10720.596176] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [10720.601728] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [10720.643675] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [10722.956225] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [10726.222226] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [10729.643832] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [10729.775247] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10731.917563] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [10732.028611] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10733.959638] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [10734.038684] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10735.994243] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [10736.086421] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10741.219990] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10741.220611] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10741.233640] Lustre: Skipped 3 previous similar messages [10741.243463] Lustre: Skipped 3 previous similar messages [10743.951540] LustreError: 534383:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [10744.016414] Lustre: server umount lustre-MDT0000 complete [10744.151861] Lustre: server umount lustre-MDT0001 complete [10746.110422] LustreError: 529797:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767325502 with bad export cookie 1549251298776826963 [10746.113381] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10746.115788] LustreError: 529797:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [10746.146807] LustreError: 534874:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [10746.150383] LustreError: 534874:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [10746.178060] Lustre: server umount lustre-OST03e8 complete [10746.225355] Lustre: server umount lustre-OST2710 complete [10761.186146] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [10787.625778] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [10795.249956] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10795.437186] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10796.214636] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10797.103930] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10802.960432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10802.981738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10803.063340] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10803.074595] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10803.113421] Lustre: lustre-MDT0000: new disk, initializing [10803.142740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10803.149708] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10804.575309] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [10811.783199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10811.808675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10811.847168] Lustre: 544026:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10811.872117] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10811.874199] Lustre: Skipped 1 previous similar message [10811.909725] Lustre: lustre-MDT0001: new disk, initializing [10811.944624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10811.959686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10811.963515] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10813.706617] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [10817.788435] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10819.395352] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10825.149744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10825.184176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10827.713673] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [10828.348482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10828.351766] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10828.388928] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10831.159727] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10832.964350] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10833.052974] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10834.714164] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10834.794897] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10838.497229] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10838.497635] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10838.501545] LustreError: Skipped 1 previous similar message [10838.511035] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10838.514064] Lustre: Skipped 1 previous similar message [10841.648088] LustreError: 547656:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [10841.650747] LustreError: 547656:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [10841.705869] Lustre: server umount lustre-OST0000 complete [10847.712262] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10847.715033] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10847.719063] Lustre: Skipped 1 previous similar message [10847.720996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10849.862205] LustreError: 547905:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [10849.864864] LustreError: 547905:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10849.916163] Lustre: server umount lustre-MDT0000 complete [10852.128076] LustreError: 544019:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767325608 with bad export cookie 1549251298776828643 [10852.130286] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10852.132345] LustreError: 544019:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [10852.296700] Lustre: server umount lustre-MDT0001 complete [10856.034732] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 22:46:50 (1767325610) [10856.699515] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [10857.398438] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 22:46:52 (1767325612) [10858.791975] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [10862.579916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10864.718127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10866.628414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10868.676061] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10869.495056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10875.003392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10875.135791] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10875.147527] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10875.211415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10875.213634] Lustre: Skipped 1 previous similar message [10876.822276] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [10881.567872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10881.618410] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10881.634807] Lustre: 550857:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10883.368738] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [10885.555915] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10887.206702] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10891.351904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10891.381827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10891.455813] Lustre: Denying initial registration attempt from nid 192.168.204.105@tcp, specified as failover [10891.459917] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [10891.463377] LustreError: 552857:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -99 [10891.465857] LustreError: 552857:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [10891.468054] LustreError: 552857:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [10891.491586] Lustre: server umount lustre-OST0000 complete [10891.493329] LustreError: 552857:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -99 [10894.304097] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10894.304628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10894.307836] Lustre: Skipped 2 previous similar messages [10894.310124] Lustre: Skipped 2 previous similar messages [10899.424469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10899.427617] Lustre: Skipped 2 previous similar messages [10900.369123] LustreError: 553272:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [10900.371451] LustreError: 553272:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [10900.409898] Lustre: server umount lustre-MDT0000 complete [10902.424280] LustreError: 550848:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767325658 with bad export cookie 1549251298776829525 [10902.426355] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10902.427886] LustreError: 550848:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [10902.567575] Lustre: server umount lustre-MDT0001 complete [10908.381937] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [10909.650939] Key type lgssc unregistered [10909.814596] LNet: 554198:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10909.819376] LNetError: 554198:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [10909.829120] LNet: Removed LNI 192.168.204.105@tcp [10910.260954] Key type .llcrypt unregistered [10910.262316] Key type ._llcrypt unregistered [10915.610446] Key type ._llcrypt registered [10915.611815] Key type .llcrypt registered [10915.653260] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [10916.124891] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10916.135476] alg: No test for adler32 (adler32-zlib) [10917.073726] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [10917.195786] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [10918.816922] Key type lgssc registered [10919.326747] Lustre: Echo OBD driver; http://www.lustre.org/ [10923.057316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10925.358380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10927.414474] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10929.435353] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10930.277350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10961.368887] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [10964.872718] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10964.884749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10966.051363] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10966.064228] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10966.158199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10967.729756] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [10972.750499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10972.791726] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10972.804669] Lustre: 560391:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10972.913544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10974.384449] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [10976.621367] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10978.131726] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10982.639829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [10982.671578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [10982.780600] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10982.869746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10985.280296] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [10987.863733] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10989.050390] LustreError: 563051:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [10989.127729] Lustre: server umount lustre-OST0000 complete [10992.113071] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10992.114667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10992.116912] Lustre: Skipped 1 previous similar message [10992.120701] Lustre: Skipped 1 previous similar message [10997.217749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10997.220932] Lustre: Skipped 1 previous similar message [10997.581119] LustreError: 563300:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [10997.585100] LustreError: 563300:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10997.618659] Lustre: server umount lustre-MDT0000 complete [10999.773566] LustreError: 560384:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767325755 with bad export cookie 2670510900185277031 [10999.776577] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10999.778105] LustreError: 560384:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [10999.851185] LustreError: 563554:0:(obd_class.h:479:obd_check_dev()) Device 7 not setup [10999.853538] LustreError: 563554:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [10999.922941] Lustre: server umount lustre-MDT0001 complete [11005.732089] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [11007.106533] Key type lgssc unregistered [11007.254656] LNet: 564229:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11007.259837] LNetError: 564229:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11007.267481] LNet: Removed LNI 192.168.204.105@tcp [11007.652541] Key type .llcrypt unregistered [11007.653900] Key type ._llcrypt unregistered [11045.379266] Key type ._llcrypt registered [11045.380325] Key type .llcrypt registered [11045.433629] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [11045.777496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11045.785239] alg: No test for adler32 (adler32-zlib) [11046.651616] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [11046.743533] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [11048.327156] Key type lgssc registered [11048.745933] Lustre: Echo OBD driver; http://www.lustre.org/ [11051.753885] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11051.760866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11052.943146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11054.568425] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11059.639176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11059.811721] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11061.315659] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11063.562895] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11065.082589] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11069.967689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11070.156916] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11073.013497] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11075.434608] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11090.399845] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11090.402635] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11090.407766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11092.326724] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.5@tcp (stopping) [11095.008275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11095.008968] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11095.016222] Lustre: Skipped 1 previous similar message [11096.382481] LustreError: 571669:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [11096.411867] Lustre: server umount lustre-MDT0000 complete [11097.438313] LustreError: 568959:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11097.445617] LustreError: 568959:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11098.192469] LustreError: 568946:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767325854 with bad export cookie 13279276327077038394 [11098.193824] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11098.198907] LustreError: 568946:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [11100.128090] LustreError: 568959:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11100.128414] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11100.138767] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11100.141946] Lustre: Skipped 2 previous similar messages [11102.560047] LustreError: 568959:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11104.331014] LustreError: 571918:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [11104.337944] LustreError: 571918:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11104.425531] Lustre: server umount lustre-MDT0001 complete [11108.151865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [11114.052234] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11123.679717] LustreError: 568276:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000003daaf470 x1853175420185344/t0(0) o250->MGC192.168.204.105@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11123.816857] LustreError: 570959:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11123.826744] LustreError: 570959:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11123.850518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11123.874141] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [11125.256316] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11127.969432] LustreError: 573062:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11127.974477] LustreError: 573062:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11130.266934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11130.417797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11132.223188] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11133.476363] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [11134.312220] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11139.081770] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [11140.493929] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11144.175788] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 3 sec [11145.696087] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11145.698600] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11145.703916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11145.706492] Lustre: Skipped 2 previous similar messages [11151.664782] LustreError: 574750:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11151.666803] LustreError: 574750:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11151.708293] Lustre: server umount lustre-OST0000 complete [11153.888506] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11153.888943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11153.893503] Lustre: Skipped 2 previous similar messages [11153.895532] Lustre: Skipped 2 previous similar messages [11159.430091] LustreError: 574999:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [11159.432536] LustreError: 574999:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11159.460253] Lustre: server umount lustre-MDT0000 complete [11161.292505] LustreError: 570971:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767325917 with bad export cookie 13279276327077045520 [11161.295251] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11161.297225] LustreError: 570971:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [11161.433514] Lustre: server umount lustre-MDT0001 complete [11166.460074] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [11167.600909] Key type lgssc unregistered [11167.728482] LNet: 575926:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11167.731413] LNetError: 575926:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11167.740473] LNet: Removed LNI 192.168.204.105@tcp [11168.051604] Key type .llcrypt unregistered [11168.053035] Key type ._llcrypt unregistered [11200.050029] Key type ._llcrypt registered [11200.051122] Key type .llcrypt registered [11200.089996] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [11200.420234] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11200.451032] alg: No test for adler32 (adler32-zlib) [11201.307395] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [11201.386904] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [11202.975165] Key type lgssc registered [11203.360441] Lustre: Echo OBD driver; http://www.lustre.org/ [11206.318138] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11206.326174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11207.487803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11208.905708] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11213.513286] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11213.688620] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11215.161220] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11217.207865] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11218.511822] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11226.446298] LustreError: 582271:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [11226.448180] LustreError: 582271:0:(obd_class.h:479:obd_check_dev()) Skipped 2 previous similar messages [11226.489550] Lustre: server umount lustre-MDT0000 complete [11226.524360] Lustre: server umount lustre-MDT0001 complete [11234.348116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11234.448860] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11234.464567] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11234.541982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11236.028779] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11240.580977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11240.610562] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11240.621628] Lustre: 583698:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11240.722387] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11242.058616] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11244.220706] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11245.777485] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11250.090266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11250.171278] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [11250.178755] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [11250.183268] LustreError: 585741:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [11250.186186] LustreError: 585741:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [11250.190278] LustreError: 585741:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [11250.215260] Lustre: server umount lustre-OST0000 complete [11250.217099] LustreError: 585741:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [11254.574723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11254.650075] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11254.728686] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11256.776894] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11258.712546] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11260.951654] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [11265.093862] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11265.118087] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11265.143572] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [11265.176121] Lustre: lustre-OST0001: new disk, initializing [11265.179100] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11265.213983] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11266.178426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11266.182993] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11266.215135] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11267.292409] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11269.667835] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [11271.650759] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11271.650943] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11271.652608] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11271.652823] LustreError: Skipped 1 previous similar message [11271.660469] Lustre: Skipped 1 previous similar message [11272.748826] LustreError: 588591:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [11272.750795] LustreError: 588591:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [11272.769814] Lustre: server umount lustre-OST0001 complete [11276.256611] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11276.256759] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11276.258802] LustreError: Skipped 1 previous similar message [11276.263338] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11276.266475] Lustre: Skipped 2 previous similar messages [11276.773949] LustreError: 586402:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11276.779783] LustreError: 586402:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11280.110986] LustreError: 588791:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [11280.113385] LustreError: 588791:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11280.157370] Lustre: server umount lustre-OST0000 complete [11281.888659] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11281.890744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11281.893159] Lustre: Skipped 2 previous similar messages [11281.893588] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11281.895877] Lustre: Skipped 1 previous similar message [11287.008172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11287.010173] Lustre: Skipped 1 previous similar message [11292.128501] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11292.130666] Lustre: Skipped 1 previous similar message [11296.223164] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11296.267687] LustreError: 589041:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [11296.270238] LustreError: 589041:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11296.308973] Lustre: server umount lustre-MDT0000 complete [11297.248138] LustreError: 584630:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11297.254269] LustreError: 584630:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11298.106298] LustreError: 583690:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767326054 with bad export cookie 10168254958045334716 [11298.109041] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11298.109636] LustreError: 583690:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [11298.244473] Lustre: server umount lustre-MDT0001 complete [11303.631666] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [11304.752178] Key type lgssc unregistered [11304.881747] LNet: 589969:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11304.884846] LNetError: 589969:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11304.891299] LNet: Removed LNI 192.168.204.105@tcp [11305.200771] Key type .llcrypt unregistered [11305.203450] Key type ._llcrypt unregistered [11323.284633] Key type ._llcrypt registered [11323.285790] Key type .llcrypt registered [11323.321775] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [11348.299964] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [11348.728379] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11348.739089] alg: No test for adler32 (adler32-zlib) [11349.602914] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [11349.690814] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [11351.271149] Key type lgssc registered [11351.678112] Lustre: Echo OBD driver; http://www.lustre.org/ [11358.041841] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11358.207921] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11358.996043] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11360.020101] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11365.628282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11365.647959] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11365.653322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11366.763468] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11366.777606] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11366.820696] Lustre: lustre-MDT0000: new disk, initializing [11366.855579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11366.863280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11368.310940] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11374.908990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11374.933459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11374.973360] Lustre: 600183:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11375.000554] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11375.002736] Lustre: Skipped 1 previous similar message [11375.045286] Lustre: lustre-MDT0001: new disk, initializing [11375.088733] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11375.102850] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11375.106185] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11376.556736] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11379.890842] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11381.177565] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11385.955144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11385.976877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11386.088959] Lustre: lustre-OST0000: new disk, initializing [11386.092114] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11386.128576] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11388.276975] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11391.387523] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11391.993234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11391.997739] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11392.041687] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11392.870895] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11392.948131] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11394.390245] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11394.487822] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11397.089882] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11397.090303] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11397.092726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11397.097049] Lustre: Skipped 1 previous similar message [11401.326699] LustreError: 603811:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [11401.373188] Lustre: server umount lustre-OST0000 complete [11405.792095] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11405.795254] LustreError: Skipped 1 previous similar message [11405.796948] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11405.803000] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11405.804750] Lustre: Skipped 1 previous similar message [11407.328300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11407.328878] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11409.281510] LustreError: 604061:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [11409.283331] LustreError: 604061:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11409.313645] Lustre: server umount lustre-MDT0000 complete [11411.211428] LustreError: 604063:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767326167 with bad export cookie 470072957290444158 [11411.214451] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11411.214988] LustreError: 604063:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [11411.280405] LustreError: 604313:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11411.282636] LustreError: 604313:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11411.348694] Lustre: server umount lustre-MDT0001 complete [11414.612291] Lustre: DEBUG MARKER: == conf-sanity test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 22:56:09 (1767326169) [11418.794377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11422.196366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11435.628794] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [11457.343126] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [11463.957882] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11464.142320] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11464.802376] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11465.701995] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11471.136681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11471.158048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11471.242340] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11471.253489] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11471.288547] Lustre: lustre-MDT0000: new disk, initializing [11471.311237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11471.317588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11472.644451] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11479.111410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11479.136099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11479.170401] Lustre: 614656:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11479.194531] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11479.196289] Lustre: Skipped 1 previous similar message [11479.231910] Lustre: lustre-MDT0001: new disk, initializing [11479.262120] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11479.274856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11479.278571] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11480.561232] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11483.725843] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11484.892642] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11489.322326] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11489.345705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11489.453410] Lustre: lustre-OST0000: new disk, initializing [11489.455312] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11489.482321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11489.868668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11489.871904] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11489.896236] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11491.357585] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11494.401794] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11495.737104] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11495.805700] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11497.129930] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11497.199311] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11500.514340] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11500.514634] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11500.518405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11500.521454] Lustre: Skipped 2 previous similar messages [11505.633287] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11505.635566] Lustre: Skipped 1 previous similar message [11512.287125] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11512.312090] LustreError: 618287:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [11512.314507] LustreError: 618287:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [11512.353490] Lustre: server umount lustre-OST0000 complete [11515.359780] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11515.361939] LustreError: Skipped 1 previous similar message [11515.363319] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11515.366771] Lustre: Skipped 1 previous similar message [11515.368560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11515.370688] Lustre: Skipped 2 previous similar messages [11528.159153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11528.194931] LustreError: 618538:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [11528.197063] LustreError: 618538:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11528.225442] Lustre: server umount lustre-MDT0000 complete [11529.677961] LustreError: 614647:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767326285 with bad export cookie 470072957290445040 [11529.681084] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11529.681180] LustreError: 614647:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [11529.796186] Lustre: server umount lustre-MDT0001 complete [11532.587966] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 22:58:07 (1767326287) [11535.426219] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 22:58:10 (1767326290) [11540.157254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11540.334366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11541.644311] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11545.567976] LustreError: 620154:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11545.573263] LustreError: 620154:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11545.967881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11547.605043] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11549.485329] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11550.765349] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11554.146048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11556.135222] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11557.896585] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11571.841791] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [11572.397435] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [11572.924980] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [11573.484474] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [11574.126599] Lustre: Failing over lustre-MDT0000 [11574.204625] LustreError: 622979:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [11574.206576] LustreError: 622979:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [11574.234092] Lustre: server umount lustre-MDT0000 complete [11575.776974] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11575.777994] LustreError: 620130:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11575.780294] Lustre: Skipped 3 previous similar messages [11575.784436] LustreError: 620130:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11578.724304] LustreError: 622678:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11580.896131] LustreError: 620135:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11580.902564] LustreError: 620135:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11586.016070] LustreError: 620135:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11586.021457] LustreError: 620135:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11590.725924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11590.781346] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11590.869637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11590.872807] Lustre: Skipped 2 previous similar messages [11590.900136] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11592.114857] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11594.077162] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11596.260891] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11596.265967] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [11596.282393] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11597.392442] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11597.921804] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11599.809713] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [11600.501260] Lustre: Failing over lustre-MDT0000 [11600.678899] LustreError: 624637:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [11600.681451] LustreError: 624637:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11600.706776] Lustre: server umount lustre-MDT0000 complete [11601.377972] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11601.378530] LustreError: 620154:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11601.381724] Lustre: Skipped 1 previous similar message [11601.386471] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11601.388135] LustreError: 620154:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11617.267254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11617.316205] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11617.447338] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11618.643016] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11619.676665] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11622.887303] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11622.889177] Lustre: Skipped 2 previous similar messages [11622.895243] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [11622.910534] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11624.054920] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11624.590531] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11628.001302] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11628.002906] Lustre: Skipped 6 previous similar messages [11633.022572] LustreError: 626401:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [11633.024621] LustreError: 626401:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [11633.053996] Lustre: server umount lustre-MDT0000 complete [11633.119870] LustreError: 621049:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11633.124920] LustreError: 621049:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 14 previous similar messages [11633.152235] Lustre: server umount lustre-MDT0001 complete [11634.458064] LustreError: 620115:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767326390 with bad export cookie 470072957290447693 [11634.460724] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11634.461827] LustreError: 620115:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [11634.540957] Lustre: server umount lustre-OST0000 complete [11642.788060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11644.224406] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11648.164632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11649.524218] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11651.321928] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11652.592325] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11656.208676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11656.315447] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11656.317424] Lustre: Skipped 3 previous similar messages [11658.093611] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11659.838611] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11668.646496] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11674.083685] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11674.084564] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11674.086021] LustreError: Skipped 1 previous similar message [11674.089185] Lustre: Skipped 2 previous similar messages [11674.089643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11674.093349] Lustre: Skipped 1 previous similar message [11677.647023] Lustre: server umount lustre-OST0000 complete [11685.205537] Lustre: server umount lustre-MDT0000 complete [11686.776094] LustreError: 628172:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767326442 with bad export cookie 470072957290448421 [11686.779017] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11686.780640] LustreError: 628172:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [11686.901095] Lustre: server umount lustre-MDT0001 complete [11691.236642] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [11692.216088] Key type lgssc unregistered [11692.334290] LNet: 632019:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11692.336194] LNetError: 632019:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11692.345258] LNet: Removed LNI 192.168.204.105@tcp [11692.608245] Key type .llcrypt unregistered [11692.609126] Key type ._llcrypt unregistered [11721.522693] Key type ._llcrypt registered [11721.523639] Key type .llcrypt registered [11721.561629] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [11721.864670] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11721.891074] alg: No test for adler32 (adler32-zlib) [11722.732513] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [11722.810684] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [11724.391134] Key type lgssc registered [11724.766378] Lustre: Echo OBD driver; http://www.lustre.org/ [11727.199599] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11727.205253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11728.348325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11729.501287] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11733.328294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11733.465120] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11734.797952] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11736.619305] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11737.837166] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11741.092039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11741.200736] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11742.995207] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11744.669585] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11753.507863] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11763.680485] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11763.681159] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11763.685378] Lustre: Skipped 1 previous similar message [11766.076705] LustreError: 639542:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [11766.078506] LustreError: 639542:0:(obd_class.h:479:obd_check_dev()) Skipped 2 previous similar messages [11766.110768] Lustre: server umount lustre-MDT0001 complete [11766.140790] Lustre: server umount lustre-MDT0000 complete [11767.419239] LustreError: 636653:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767326523 with bad export cookie 711548674131010973 [11767.422581] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11767.423361] LustreError: 636653:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [11767.444433] LustreError: 640007:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [11767.446378] LustreError: 640007:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [11767.489251] Lustre: server umount lustre-OST0000 complete [11773.587784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11773.721446] LustreError: 641129:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11773.726789] LustreError: 641129:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11773.742503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11774.996444] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11778.767211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11778.885100] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11780.130967] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11781.845228] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11783.010923] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11786.429139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11788.379222] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11790.135211] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11798.822248] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11809.248419] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11809.249041] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11809.252665] Lustre: Skipped 1 previous similar message [11809.254891] Lustre: Skipped 1 previous similar message [11813.552240] LustreError: 644022:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [11813.554553] LustreError: 644022:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [11813.589407] Lustre: server umount lustre-MDT0000 complete [11813.640862] Lustre: server umount lustre-MDT0001 complete [11815.016474] LustreError: 643132:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767326571 with bad export cookie 711548674131012072 [11815.020895] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11815.088996] Lustre: server umount lustre-OST0000 complete [11822.651710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11822.822222] LustreError: 645830:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11822.844273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11822.846542] Lustre: Skipped 1 previous similar message [11824.070952] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11828.115543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11828.191916] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11828.193916] Lustre: Skipped 1 previous similar message [11829.634549] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11831.440200] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11832.704750] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11836.300512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11838.354684] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11840.138973] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11849.769104] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11853.792345] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11853.794465] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11853.798203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11854.819053] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11857.774103] LustreError: 648481:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [11857.775620] LustreError: 648481:0:(obd_class.h:479:obd_check_dev()) Skipped 15 previous similar messages [11857.810788] Lustre: server umount lustre-OST0000 complete [11859.936170] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11859.936760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11859.940369] Lustre: Skipped 2 previous similar messages [11859.942010] Lustre: Skipped 2 previous similar messages [11865.403938] LustreError: 648731:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [11865.405684] LustreError: 648731:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [11865.428376] Lustre: server umount lustre-MDT0000 complete [11866.889454] LustreError: 648381:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767326623 with bad export cookie 711548674131013080 [11866.890042] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11866.893780] LustreError: 648381:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [11867.003105] Lustre: server umount lustre-MDT0001 complete [11871.206504] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [11872.190755] Key type lgssc unregistered [11872.320436] LNet: 649655:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11872.323172] LNetError: 649655:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11872.330431] LNet: Removed LNI 192.168.204.105@tcp [11872.612748] Key type .llcrypt unregistered [11872.614082] Key type ._llcrypt unregistered [11899.619422] Key type ._llcrypt registered [11899.620720] Key type .llcrypt registered [11899.654739] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [11900.038359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11900.049633] alg: No test for adler32 (adler32-zlib) [11900.896061] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [11900.972964] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [11902.551097] Key type lgssc registered [11902.865297] Lustre: Echo OBD driver; http://www.lustre.org/ [11905.196041] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11905.202110] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11905.205250] LustreError: 654220:0:(osd_handler.c:8561:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11905.213868] LustreError: 654220:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000-osd failed (-22) [11905.215799] LustreError: 654220:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11905.218205] LustreError: 654220:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11905.220888] LustreError: 654220:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [11908.919913] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [11908.923304] LustreError: 654811:0:(osd_handler.c:8561:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11908.932101] LustreError: 654811:0:(obd_config.c:783:class_setup()) setup lustre-OST0000-osd failed (-22) [11908.935437] LustreError: 654811:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11908.938398] LustreError: 654811:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11908.941743] LustreError: 654811:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [11916.363171] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [11917.296643] Key type lgssc unregistered [11917.409278] LNet: 656035:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11917.411437] LNetError: 656035:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [11917.419281] LNet: Removed LNI 192.168.204.105@tcp [11917.689528] Key type .llcrypt unregistered [11917.690771] Key type ._llcrypt unregistered [11926.914677] Key type ._llcrypt registered [11926.915553] Key type .llcrypt registered [11926.953457] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [11949.491586] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [11949.850721] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11949.868879] alg: No test for adler32 (adler32-zlib) [11950.715520] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [11950.790716] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [11952.367156] Key type lgssc registered [11952.677149] Lustre: Echo OBD driver; http://www.lustre.org/ [11958.475979] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11958.919791] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11959.666962] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11960.461886] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11965.741802] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11965.763605] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11965.768605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11966.858577] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11966.872651] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11966.914147] Lustre: lustre-MDT0000: new disk, initializing [11966.940807] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11966.948423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11968.308029] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11974.717747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11974.741891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11974.780476] Lustre: 665043:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11974.804786] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11974.806722] Lustre: Skipped 1 previous similar message [11974.845519] Lustre: lustre-MDT0001: new disk, initializing [11974.880366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11974.894393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11974.898325] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11976.283967] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11979.216153] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11980.310258] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11984.395857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [11984.425018] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [11984.510255] Lustre: lustre-OST0000: new disk, initializing [11984.511958] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11984.536329] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11986.264321] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [11986.627847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11986.630445] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11986.668593] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11989.307308] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11990.569869] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11990.648143] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11992.005547] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11992.086223] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11997.154451] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11997.154504] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11997.156406] LustreError: Skipped 1 previous similar message [11997.161219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11997.163962] Lustre: Skipped 1 previous similar message [12002.274068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12002.276921] Lustre: Skipped 1 previous similar message [12006.879113] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12006.901197] LustreError: 668669:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [12006.938610] Lustre: server umount lustre-OST0000 complete [12010.975874] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12010.979434] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12010.983244] Lustre: Skipped 1 previous similar message [12010.985321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12012.513030] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12014.591786] LustreError: 668920:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [12014.593841] LustreError: 668920:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12014.630038] Lustre: server umount lustre-MDT0000 complete [12016.124121] LustreError: 665034:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767326772 with bad export cookie 4421149535010971714 [12016.126497] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12016.127242] LustreError: 665034:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [12016.188713] LustreError: 669170:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12016.190581] LustreError: 669170:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [12016.238030] Lustre: server umount lustre-MDT0001 complete [12018.953196] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [12019.443208] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 23:06:14 (1767326774) [12022.701311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12022.858627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12024.074814] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [12027.871973] LustreError: 670081:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12027.876563] LustreError: 670081:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12027.946755] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12028.076613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12029.363697] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [12031.125277] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12032.314769] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12035.493812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12037.220965] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [12038.798340] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12043.759066] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12043.780495] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12043.832574] Lustre: lustre-OST0001: new disk, initializing [12043.834445] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12043.859765] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12043.862291] Lustre: Skipped 1 previous similar message [12044.934889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12044.938175] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12044.975389] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12045.707196] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [12049.035964] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [12053.475627] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12053.479891] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [12053.482038] Lustre: Skipped 3 previous similar messages [12056.942085] LustreError: 674202:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [12056.944861] LustreError: 674202:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [12056.963797] Lustre: server umount lustre-OST0001 complete [12058.596205] LustreError: 672064:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12058.601817] LustreError: 672064:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12063.717378] LustreError: 673648:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12063.724841] LustreError: 673648:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12068.834132] LustreError: 673648:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12068.838927] LustreError: 673648:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12069.344486] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12069.347184] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12069.351338] Lustre: Skipped 1 previous similar message [12069.353385] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12069.354960] Lustre: Skipped 1 previous similar message [12073.954084] LustreError: 672063:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12073.955914] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12073.958604] LustreError: 672063:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12075.053181] LustreError: 674427:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12075.054947] LustreError: 674427:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12075.085551] Lustre: server umount lustre-OST0000 complete [12079.072960] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12079.073554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12079.077137] Lustre: Skipped 2 previous similar messages [12079.078853] Lustre: Skipped 2 previous similar messages [12082.716353] Lustre: server umount lustre-MDT0000 complete [12084.193024] LustreError: 670080:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12084.200102] LustreError: 670080:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12084.276263] LustreError: 670060:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767326840 with bad export cookie 4421149535010972589 [12084.279696] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12084.280858] LustreError: 670060:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [12090.430249] LustreError: 674928:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [12090.431788] LustreError: 674928:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [12090.485801] Lustre: server umount lustre-MDT0001 complete [12094.961325] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [12095.970182] Key type lgssc unregistered [12096.089397] LNet: 675604:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12096.091960] LNetError: 675604:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12096.100306] LNet: Removed LNI 192.168.204.105@tcp [12096.359263] Key type .llcrypt unregistered [12096.360487] Key type ._llcrypt unregistered [12111.975635] Key type ._llcrypt registered [12111.976537] Key type .llcrypt registered [12112.012031] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [12135.702196] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [12136.012047] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12136.051047] alg: No test for adler32 (adler32-zlib) [12136.908401] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [12136.985271] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [12138.567153] Key type lgssc registered [12138.892579] Lustre: Echo OBD driver; http://www.lustre.org/ [12144.486377] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12144.804909] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12145.443777] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12146.336327] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12151.307543] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12151.326819] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12151.331411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12152.403531] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12152.413461] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12152.445756] Lustre: lustre-MDT0000: new disk, initializing [12152.468119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12152.474386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12153.622576] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [12159.598051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12159.617543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12159.649496] Lustre: 685818:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12159.671683] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12159.673230] Lustre: Skipped 1 previous similar message [12159.705867] Lustre: lustre-MDT0001: new disk, initializing [12159.736813] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12159.747453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12159.750436] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12160.878649] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [12163.808152] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12164.990823] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12169.181646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12169.208643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12169.316971] Lustre: lustre-OST0000: new disk, initializing [12169.319946] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12169.352914] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12171.185667] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [12172.920325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12172.923403] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12172.947763] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12174.136342] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12175.392189] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12175.466032] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12176.701803] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12176.767748] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12178.401697] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12178.402487] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12178.403060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12178.403729] LustreError: Skipped 1 previous similar message [12178.408141] Lustre: Skipped 1 previous similar message [12179.436071] LustreError: 689445:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [12179.472045] Lustre: server umount lustre-OST0000 complete [12183.521195] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12183.521903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12183.525655] Lustre: Skipped 1 previous similar message [12183.528117] Lustre: Skipped 1 previous similar message [12188.640099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12188.641665] Lustre: Skipped 2 previous similar messages [12193.759966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12193.762023] Lustre: Skipped 1 previous similar message [12195.295143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12195.326772] LustreError: 689694:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [12195.328346] LustreError: 689694:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12195.356855] Lustre: server umount lustre-MDT0000 complete [12196.835307] LustreError: 685808:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767326953 with bad export cookie 6872277251373724787 [12196.838591] LustreError: 685808:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [12196.838841] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12196.913725] LustreError: 689944:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12196.916774] LustreError: 689944:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [12196.979313] Lustre: server umount lustre-MDT0001 complete [12199.715333] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 23:09:14 (1767326954) [12203.496948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [12207.082254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12207.207938] Lustre: lustre-MDT0000: new disk, initializing [12207.236354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12208.440187] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [12212.135968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12212.245867] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12213.507083] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [12215.152245] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12216.237490] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12217.147771] LustreError: 691573:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [12217.149809] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12217.150564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12217.151271] LustreError: 691573:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [12217.154384] Lustre: Skipped 1 previous similar message [12217.159248] LustreError: 691573:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -5 [12223.291657] LustreError: 693127:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [12223.293904] LustreError: 693127:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [12223.315790] Lustre: server umount lustre-MDT0000 complete [12224.814976] LustreError: 691524:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767326981 with bad export cookie 6872277251373725669 [12224.817782] LustreError: 691524:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [12224.817786] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12224.922597] Lustre: server umount lustre-MDT0001 complete [12227.806687] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 23:09:42 (1767326982) [12230.901932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12231.035848] LustreError: 694291:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12231.041535] LustreError: 694291:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12231.056333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12232.259457] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [12236.046568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12237.399264] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [12239.090655] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12240.205384] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12243.551331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12245.420123] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [12247.186466] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12260.264138] Lustre: Permanently deactivating lustre-OST0000 [12260.269140] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12270.063563] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12271.128092] LustreError: 697532:0:(mgs_llog.c:1942:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12271.131566] LustreError: 697532:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [12271.134919] LustreError: 697532:0:(mgs_handler.c:1171:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12271.773680] LustreError: 697581:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [12271.775943] LustreError: 697581:0:(mgs_handler.c:1171:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12278.511092] LustreError: 697679:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12278.513645] LustreError: 697679:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [12278.554996] Lustre: server umount lustre-OST0000 complete [12279.542327] LustreError: 697831:0:(mgs_llog.c:1997:mgs_replace_nids()) Only MGS is allowed to be started [12279.544890] LustreError: 697831:0:(mgs_handler.c:1171:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12282.335873] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12282.338693] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12282.343346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12282.346268] Lustre: Skipped 3 previous similar messages [12290.019427] LustreError: 697976:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [12290.022114] LustreError: 697976:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12290.048170] Lustre: server umount lustre-MDT0000 complete [12291.602713] LustreError: 694271:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767327047 with bad export cookie 6872277251373726201 [12291.603135] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12291.606805] LustreError: 694271:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 1 previous similar message [12301.840807] Lustre: server umount lustre-MDT0001 complete [12306.645385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12307.882413] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [12308.806798] LustreError: 699404:0:(mgs_handler.c:1171:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12310.906602] Lustre: 699742:0:(mgs_llog.c:1709:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.105@tcp [12312.799536] LustreError: 700032:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [12312.801276] LustreError: 700032:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [12312.821758] Lustre: server umount MGS complete [12318.854252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12318.983093] LustreError: 701044:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12318.988040] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12318.991882] Lustre: Skipped 1 previous similar message [12319.008516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12319.011163] Lustre: Skipped 2 previous similar messages [12320.192717] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [12324.038527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12324.126587] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12325.395580] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [12327.145044] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12328.354262] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12328.974707] Lustre: Permanently reactivating lustre-OST0000 [12328.979405] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12328.981171] Lustre: Skipped 2 previous similar messages [12355.039112] LustreError: 702839:0:(osp_dev.c:692:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12359.522036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12359.639021] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12359.641127] Lustre: Skipped 1 previous similar message [12361.456710] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [12363.174126] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12375.105467] LustreError: 704861:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12375.108120] LustreError: 704861:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [12375.145310] Lustre: server umount lustre-OST0000 complete [12380.129318] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12380.130115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12380.134808] Lustre: Skipped 2 previous similar messages [12380.136638] Lustre: Skipped 3 previous similar messages [12386.299343] Lustre: server umount lustre-MDT0000 complete [12387.758069] LustreError: 702583:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767327143 with bad export cookie 6872277251373727146 [12387.760351] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12387.762167] LustreError: 702583:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [12397.068532] Lustre: server umount lustre-MDT0001 complete [12401.286312] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [12402.227066] Key type lgssc unregistered [12402.354340] LNet: 706040:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12402.357383] LNetError: 706040:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12402.365324] LNet: Removed LNI 192.168.204.105@tcp [12402.631350] Key type .llcrypt unregistered [12402.632843] Key type ._llcrypt unregistered [12412.218597] Key type ._llcrypt registered [12412.219644] Key type .llcrypt registered [12412.252647] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [12434.612979] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [12435.001764] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12435.010389] alg: No test for adler32 (adler32-zlib) [12435.855614] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [12435.932582] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [12437.511131] Key type lgssc registered [12437.824262] Lustre: Echo OBD driver; http://www.lustre.org/ [12443.612905] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12443.954375] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12444.439792] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12445.245680] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12450.113405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12450.131814] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12450.137356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12451.232236] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12451.243749] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12451.283077] Lustre: lustre-MDT0000: new disk, initializing [12451.308256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12451.314860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12452.597620] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [12458.616879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12458.646585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12458.682284] Lustre: 715047:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12458.704933] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12458.706525] Lustre: Skipped 1 previous similar message [12458.744916] Lustre: lustre-MDT0001: new disk, initializing [12458.778843] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12458.791915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12458.794869] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12460.124153] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [12463.241193] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12464.543195] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12469.086611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12469.115867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12469.219561] Lustre: lustre-OST0000: new disk, initializing [12469.221428] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12469.250628] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12471.233111] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [12474.348429] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12475.785886] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12477.946092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12477.949453] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12477.982259] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12478.916783] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [12480.343148] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12480.439329] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12483.041040] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12483.043974] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12483.047119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12483.050162] Lustre: Skipped 1 previous similar message [12487.160560] LustreError: 718698:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [12487.199257] Lustre: server umount lustre-OST0000 complete [12489.696037] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12489.697986] LustreError: Skipped 1 previous similar message [12489.699163] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12489.703255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12489.705621] Lustre: Skipped 1 previous similar message [12493.280137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12493.280828] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12494.916107] LustreError: 718948:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [12494.919152] LustreError: 718948:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12494.953842] Lustre: server umount lustre-MDT0000 complete [12496.599412] LustreError: 715038:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767327252 with bad export cookie 12497459516609667565 [12496.603573] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12496.604548] LustreError: 715038:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 3 previous similar messages [12496.674055] LustreError: 719197:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12496.675949] LustreError: 719197:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [12496.742434] Lustre: server umount lustre-MDT0001 complete [12499.485391] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12500.050763] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 23:14:14 (1767327254) [12504.266646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12504.433276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12505.752738] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [12509.577198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12509.685644] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12510.970607] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [12512.679741] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12513.858651] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12517.202303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12519.144786] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [12520.853905] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12521.455861] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12532.138539] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12534.752996] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12534.754613] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12534.758488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12534.761244] Lustre: Skipped 1 previous similar message [12534.763275] Lustre: Skipped 1 previous similar message [12539.566166] LustreError: 723097:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12539.568212] LustreError: 723097:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [12539.608036] Lustre: server umount lustre-OST0000 complete [12544.992419] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12544.993238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12544.997319] Lustre: Skipped 1 previous similar message [12547.389176] LustreError: 723348:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [12547.390866] LustreError: 723348:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12547.416607] Lustre: server umount lustre-MDT0000 complete [12548.913565] LustreError: 722255:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767327305 with bad export cookie 12497459516609668433 [12548.917131] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12548.920383] LustreError: 722255:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [12549.038658] Lustre: server umount lustre-MDT0001 complete [12553.253299] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [12554.283555] Key type lgssc unregistered [12554.410558] LNet: 724272:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12554.414500] LNetError: 724272:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12554.426408] LNet: Removed LNI 192.168.204.105@tcp [12554.738220] Key type .llcrypt unregistered [12554.739140] Key type ._llcrypt unregistered [12563.449800] Key type ._llcrypt registered [12563.450695] Key type .llcrypt registered [12563.490757] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [12563.591231] Key type .llcrypt unregistered [12563.592052] Key type ._llcrypt unregistered [12587.791617] Key type ._llcrypt registered [12587.792806] Key type .llcrypt registered [12587.831275] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [12588.266445] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12588.308446] alg: No test for adler32 (adler32-zlib) [12589.159237] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [12589.243911] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [12590.823230] Key type lgssc registered [12591.183025] Lustre: Echo OBD driver; http://www.lustre.org/ [12593.653620] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12593.659029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12594.800811] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12595.974930] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [12599.748656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12599.861709] LustreError: 729729:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12599.865239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12599.865723] LustreError: 729729:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12601.526329] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [12603.099139] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12603.936870] LustreError: 729733:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12603.940903] LustreError: 729733:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12603.941364] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12606.352730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12606.476221] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12607.715298] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [12616.676863] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12616.677311] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12616.680818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12616.683491] Lustre: Skipped 1 previous similar message [12620.524232] LustreError: 732171:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [12620.562694] Lustre: server umount lustre-OST0000 complete [12624.865236] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12624.867149] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12624.869852] Lustre: Skipped 1 previous similar message [12624.871538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12626.912229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12626.912586] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12628.157041] LustreError: 732421:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [12628.158966] LustreError: 732421:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12628.182692] Lustre: server umount lustre-MDT0000 complete [12629.581056] LustreError: 729713:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767327385 with bad export cookie 6001906456874560799 [12629.584228] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12629.649049] LustreError: 732670:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12629.651296] LustreError: 732670:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [12629.700260] Lustre: server umount lustre-MDT0001 complete [12633.928435] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [12634.900939] Key type lgssc unregistered [12635.024339] LNet: 733345:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12635.027374] LNetError: 733345:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12635.036232] LNet: Removed LNI 192.168.204.105@tcp [12635.294539] Key type .llcrypt unregistered [12635.295657] Key type ._llcrypt unregistered [12661.231513] Key type ._llcrypt registered [12661.232466] Key type .llcrypt registered [12661.267799] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [12661.627359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12661.672121] alg: No test for adler32 (adler32-zlib) [12662.522808] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [12662.600086] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [12664.175130] Key type lgssc registered [12664.526510] Lustre: Echo OBD driver; http://www.lustre.org/ [12666.989637] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12666.996053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12682.399262] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12687.519166] LustreError: 737765:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.105@tcp: failed processing log, type 4: rc = -110 [12714.143189] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12716.078398] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [12717.909135] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12721.382875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12721.574748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12721.597911] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12722.937831] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [12726.753503] LustreError: 737792:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12726.759122] LustreError: 737792:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12727.034309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12727.167940] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12728.488964] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [12730.428986] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12731.730704] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12734.433033] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12734.437783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12734.439620] Lustre: Skipped 1 previous similar message [12739.556243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12739.558382] Lustre: Skipped 1 previous similar message [12740.079888] LustreError: 740432:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [12740.117083] Lustre: server umount lustre-OST0000 complete [12742.623886] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12742.626385] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12742.629924] Lustre: Skipped 1 previous similar message [12742.631620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12744.672094] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12744.672901] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12747.839701] LustreError: 740681:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [12747.843480] LustreError: 740681:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12747.873585] Lustre: server umount lustre-MDT0000 complete [12749.462147] LustreError: 737773:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767327505 with bad export cookie 13216189671492742983 [12749.465868] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12749.467446] LustreError: 737773:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [12749.536453] LustreError: 740933:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12749.538418] LustreError: 740933:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [12749.586203] Lustre: server umount lustre-MDT0001 complete [12754.189922] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [12755.212898] Key type lgssc unregistered [12755.351454] LNet: 741608:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12755.354135] LNetError: 741608:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12755.363439] LNet: Removed LNI 192.168.204.105@tcp [12755.692095] Key type .llcrypt unregistered [12755.693204] Key type ._llcrypt unregistered [12785.051085] Key type ._llcrypt registered [12785.052067] Key type .llcrypt registered [12785.094650] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [12785.673344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12785.714506] alg: No test for adler32 (adler32-zlib) [12786.591944] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [12786.690092] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [12788.279206] Key type lgssc registered [12788.706840] Lustre: Echo OBD driver; http://www.lustre.org/ [12791.339276] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12791.345601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12792.502683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12793.746967] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [12797.492638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12797.597225] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12798.794430] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [12800.479814] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12801.575267] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12804.916171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12805.031883] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12806.945433] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [12808.667327] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12817.317852] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12822.497179] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12822.499408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12822.501388] Lustre: Skipped 2 previous similar messages [12822.503881] Lustre: Skipped 2 previous similar messages [12824.419903] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.5@tcp (stopping) [12826.557788] LustreError: 748756:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [12826.580668] Lustre: server umount lustre-MDT0000 complete [12827.282902] LustreError: 746968:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.5@tcp arrived at 1767327583 with bad export cookie 8576307400396221632 [12827.287285] LustreError: 746968:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [12827.616925] LustreError: 746974:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12827.623043] LustreError: 746974:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [12827.885638] LustreError: 746968:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767327584 with bad export cookie 8576307400396221009 [12827.889763] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12827.893137] LustreError: 746968:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 2 previous similar messages [12832.736706] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12832.736880] LustreError: 746974:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12832.739706] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12832.745582] LustreError: 746974:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12832.753386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12833.964124] LustreError: 748960:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12833.966653] LustreError: 748960:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [12834.011830] Lustre: server umount lustre-OST0000 complete [12836.760125] LustreError: 749409:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [12836.762062] LustreError: 749409:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12836.811173] Lustre: server umount lustre-MDT0001 complete [12840.996786] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [12841.940680] Key type lgssc unregistered [12842.070337] LNet: 750085:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12842.073408] LNetError: 750085:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12842.081445] LNet: Removed LNI 192.168.204.105@tcp [12842.376391] Key type .llcrypt unregistered [12842.377706] Key type ._llcrypt unregistered [12869.379654] Key type ._llcrypt registered [12869.380547] Key type .llcrypt registered [12869.420816] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [12869.717048] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12869.722748] alg: No test for adler32 (adler32-zlib) [12870.560657] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [12870.642593] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [12872.215126] Key type lgssc registered [12872.546796] Lustre: Echo OBD driver; http://www.lustre.org/ [12875.038780] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12875.044517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12876.193522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12877.478571] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [12881.538534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12881.667363] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12882.996073] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [12884.658915] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12885.774967] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12888.987398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [12889.103343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12890.842664] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [12892.482742] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12901.410737] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12906.466054] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12906.466610] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12906.468994] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12906.471963] Lustre: Skipped 1 previous similar message [12906.475649] Lustre: Skipped 1 previous similar message [12907.875673] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.5@tcp (stopping) [12911.585276] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12911.586893] Lustre: Skipped 1 previous similar message [12916.706242] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12916.708914] Lustre: Skipped 2 previous similar messages [12918.239107] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12918.279860] LustreError: 757231:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [12918.302801] Lustre: server umount lustre-MDT0001 complete [12921.824730] LustreError: 754533:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12921.825024] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12921.825336] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12921.826303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12921.826307] Lustre: Skipped 1 previous similar message [12921.831980] LustreError: 754533:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12926.061107] LustreError: 757436:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [12926.063432] LustreError: 757436:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [12926.100572] Lustre: server umount lustre-OST0000 complete [12926.944809] LustreError: 754533:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12927.749837] LustreError: 757685:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [12927.752940] LustreError: 757685:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [12927.817113] Lustre: server umount lustre-MDT0000 complete [12933.973458] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [12935.030139] Key type lgssc unregistered [12935.162554] LNet: 758560:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12935.166614] LNetError: 758560:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [12935.173307] LNet: Removed LNI 192.168.204.105@tcp [12935.489580] Key type .llcrypt unregistered [12935.490622] Key type ._llcrypt unregistered [12946.950965] Key type ._llcrypt registered [12946.951853] Key type .llcrypt registered [12946.985348] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [12969.214663] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [12969.684825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12969.737531] alg: No test for adler32 (adler32-zlib) [12970.601095] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [12970.693819] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [12972.271187] Key type lgssc registered [12972.612952] Lustre: Echo OBD driver; http://www.lustre.org/ [12979.144635] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12979.348398] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12979.933648] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [12980.777994] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13002.070597] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [13008.398933] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13012.769163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13012.789283] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13012.794486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13013.884280] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13013.894969] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13013.931567] Lustre: lustre-MDT0000: new disk, initializing [13013.954785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13013.962859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13015.184404] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [13018.556738] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13022.634085] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13022.656444] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13022.755731] Lustre: lustre-OST0000: new disk, initializing [13022.758512] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13022.760018] Lustre: Skipped 1 previous similar message [13022.781447] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13023.614719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13023.618541] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13023.629357] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13024.526152] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [13027.560662] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13031.451310] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13035.626175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13035.647307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13035.675748] Lustre: 771627:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13035.700616] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13035.736392] Lustre: lustre-MDT0001: new disk, initializing [13035.777929] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13035.790772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13035.794586] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13037.065440] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [13040.514481] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13040.581055] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13041.823725] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13043.927456] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 2 sec [13046.628704] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.5@tcp (stopping) [13048.801755] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13048.801844] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13048.804499] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13048.807766] Lustre: Skipped 1 previous similar message [13051.740962] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.5@tcp (stopping) [13051.743503] Lustre: Skipped 1 previous similar message [13051.774460] LustreError: 775946:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [13051.795658] Lustre: server umount lustre-MDT0001 complete [13059.040972] LustreError: 773582:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13059.044754] LustreError: 773582:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [13064.160566] LustreError: 771639:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13064.160774] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13064.170448] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13069.281337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13069.281805] LustreError: 771639:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13070.124103] LustreError: 776220:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [13070.125971] LustreError: 776220:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [13070.162217] Lustre: server umount lustre-OST0000 complete [13078.078223] LustreError: 776470:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [13078.080130] LustreError: 776470:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [13078.139924] Lustre: server umount lustre-MDT0000 complete [13083.763752] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [13084.865504] Key type lgssc unregistered [13084.999421] LNet: 777346:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13085.001791] LNetError: 777346:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [13085.012416] LNet: Removed LNI 192.168.204.105@tcp [13085.335947] Key type .llcrypt unregistered [13085.337109] Key type ._llcrypt unregistered [13105.190903] Key type ._llcrypt registered [13105.191732] Key type .llcrypt registered [13105.230524] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [13105.333423] Key type .llcrypt unregistered [13105.335053] Key type ._llcrypt unregistered [13129.835828] Key type ._llcrypt registered [13129.836676] Key type .llcrypt registered [13129.874609] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [13130.286828] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13130.347652] alg: No test for adler32 (adler32-zlib) [13131.210203] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [13131.280938] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [13132.855174] Key type lgssc registered [13133.261224] Lustre: Echo OBD driver; http://www.lustre.org/ [13138.957399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13142.535670] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13144.781324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13148.330263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13148.348404] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13148.354176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13149.448674] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13149.462570] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13149.502638] Lustre: lustre-MDT0000: new disk, initializing [13149.527280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13149.534656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13150.877341] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [13156.283023] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13156.303558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13156.337475] Lustre: 785563:0:(mgs_llog.c:1348:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13156.368380] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13156.371445] Lustre: Skipped 1 previous similar message [13156.419398] Lustre: lustre-MDT0001: new disk, initializing [13156.456489] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13156.470030] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13156.474862] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13157.779390] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [13160.912514] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13162.097311] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13165.591558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13165.619767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13165.738658] Lustre: lustre-OST0000: new disk, initializing [13165.741331] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13165.777930] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13167.744498] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [13170.920944] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13171.192350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13171.196091] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13171.234394] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13181.411178] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13181.412750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13181.417328] Lustre: Skipped 2 previous similar messages [13181.420148] Lustre: Skipped 2 previous similar messages [13184.766297] LustreError: 788438:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [13184.796735] Lustre: server umount lustre-MDT0000 complete [13186.419050] LustreError: 785554:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767327942 with bad export cookie 14968768394439402130 [13186.423117] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13186.423403] LustreError: 785554:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [13186.491535] LustreError: 788688:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [13186.494459] LustreError: 788688:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [13186.540121] Lustre: server umount lustre-MDT0001 complete [13196.914864] LustreError: 788888:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [13196.917243] LustreError: 788888:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [13196.963991] Lustre: server umount lustre-OST0000 complete [13200.472963] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 23:25:55 (1767327955) [13201.102771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13205.634458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13205.828432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13207.128080] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [13211.104631] LustreError: 790100:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13211.109407] LustreError: 790100:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [13211.298563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13211.441473] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13212.846796] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [13214.760496] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13215.985170] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13219.601463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13219.622177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13219.688294] Lustre: Found index 0 for lustre-OST0000, updating log [13219.752477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13221.458841] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [13223.204157] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13231.014338] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13236.194140] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13236.194852] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13236.195247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13236.198185] Lustre: Skipped 1 previous similar message [13237.217912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13237.220795] Lustre: Skipped 2 previous similar messages [13240.815143] LustreError: 792762:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [13240.818168] LustreError: 792762:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [13240.858187] Lustre: server umount lustre-OST0000 complete [13241.822964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13243.873353] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13243.879583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13243.882019] Lustre: Skipped 1 previous similar message [13248.992384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13248.995689] Lustre: Skipped 1 previous similar message [13257.183123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13257.220937] LustreError: 793067:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [13257.223397] LustreError: 793067:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [13257.253605] Lustre: server umount lustre-MDT0000 complete [13258.856298] LustreError: 792101:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767328015 with bad export cookie 14968768394439403509 [13258.860193] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13258.862558] LustreError: 792101:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [13258.998256] Lustre: server umount lustre-MDT0001 complete [13261.801536] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 23:26:56 (1767328016) [13262.437128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13262.793278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13263.152463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13263.506470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13263.849657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13264.175179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13264.527340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13264.866282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13265.208144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13265.551313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13265.904699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13266.272194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13266.639239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13266.998462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13267.346321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13267.707262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13268.074400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13268.438399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13268.806222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13269.173378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13269.527059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13269.887722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13270.242151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13270.590368] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13270.965163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13271.321892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13271.665375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13272.016091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13272.358134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13272.689642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13273.032690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13273.383462] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13273.720487] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13274.074551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13274.410967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13274.745038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13275.083623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13275.433588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13275.759964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13276.086752] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13276.449366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13276.776821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13277.105565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13277.441403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13277.763850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13278.093302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13278.435168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13278.789492] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13279.124467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13279.456436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13279.788252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13280.799239] LNet: 1 local NIs in recovery (showing 1): 192.168.204.105@tcp [13284.488696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13284.648119] LustreError: 797086:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13284.678176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13286.069439] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [13289.953056] LustreError: 797086:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13290.138531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13291.659311] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [13293.518470] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13294.754828] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13298.277637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13298.308617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [13298.391090] Lustre: Found index 0 for lustre-OST0000, updating log [13300.326989] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug -1 all [13302.151247] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13306.848515] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13306.851039] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13306.855165] Lustre: Skipped 1 previous similar message [13306.857422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13306.859127] Lustre: Skipped 2 previous similar messages [13308.908279] LustreError: 799750:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [13308.910103] LustreError: 799750:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [13308.951740] Lustre: server umount lustre-OST0000 complete [13310.943972] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13310.946184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13316.694245] Lustre: server umount lustre-MDT0000 complete [13317.088781] LustreError: 797086:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13317.093329] LustreError: 797086:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13318.250335] LustreError: 797067:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1767328074 with bad export cookie 14968768394439404580 [13318.253532] LustreError: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13318.253594] LustreError: 797067:0:(ldlm_lockd.c:2563:ldlm_cancel_handler()) Skipped 4 previous similar messages [13318.379758] Lustre: server umount lustre-MDT0001 complete [13319.339031] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13328.672974] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [13343.264082] LNet: 1 local NIs in recovery (showing 1): 192.168.204.105@tcp [13350.034858] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [13357.293753] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13357.735367] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13358.458989] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13359.298097] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13361.618910] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 23:28:36 (1767328116) [13369.116447] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [13370.208573] Key type lgssc unregistered [13370.359697] LNet: 809477:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13370.362858] LNetError: 809477:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [13485.094232] LNet: Removed LNI 192.168.204.105@tcp [13485.425287] Key type .llcrypt unregistered [13485.427950] Key type ._llcrypt unregistered [13510.151350] Key type ._llcrypt registered [13510.153058] Key type .llcrypt registered [13510.200882] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [13510.712780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13510.729702] alg: No test for adler32 (adler32-zlib) [13511.583942] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [13511.676720] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [13513.255140] Key type lgssc registered [13513.673196] Lustre: Echo OBD driver; http://www.lustre.org/ [13515.271750] LNet: Added LNI 192.168.204.105@tcp100 [8/256/0/180] [13515.591288] LNet: Added LNI 192.168.204.105@tcp101 [8/256/0/180] [13515.905385] LNet: Added LNI 192.168.204.105@tcp102 [8/256/0/180] [13516.204756] LNet: Added LNI 192.168.204.105@tcp103 [8/256/0/180] [13516.510268] LNet: Added LNI 192.168.204.105@tcp104 [8/256/0/180] [13516.811567] LNet: Added LNI 192.168.204.105@tcp105 [8/256/0/180] [13517.117514] LNet: Added LNI 192.168.204.105@tcp106 [8/256/0/180] [13517.407234] LNet: Added LNI 192.168.204.105@tcp107 [8/256/0/180] [13517.690846] LNet: Added LNI 192.168.204.105@tcp108 [8/256/0/180] [13517.979179] LNet: Added LNI 192.168.204.105@tcp109 [8/256/0/180] [13518.272478] LNet: Added LNI 192.168.204.105@tcp110 [8/256/0/180] [13518.577421] LNet: Added LNI 192.168.204.105@tcp111 [8/256/0/180] [13518.881776] LNet: Added LNI 192.168.204.105@tcp112 [8/256/0/180] [13519.172345] LNet: Added LNI 192.168.204.105@tcp113 [8/256/0/180] [13519.484209] LNet: Added LNI 192.168.204.105@tcp114 [8/256/0/180] [13519.794379] LNet: Added LNI 192.168.204.105@tcp115 [8/256/0/180] [13520.097420] LNet: Added LNI 192.168.204.105@tcp116 [8/256/0/180] [13520.395053] LNet: Added LNI 192.168.204.105@tcp117 [8/256/0/180] [13520.696396] LNet: Added LNI 192.168.204.105@tcp118 [8/256/0/180] [13521.001506] LNet: Added LNI 192.168.204.105@tcp119 [8/256/0/180] [13521.307483] LNet: Added LNI 192.168.204.105@tcp120 [8/256/0/180] [13521.603852] LNet: Added LNI 192.168.204.105@tcp121 [8/256/0/180] [13521.892687] LNet: Added LNI 192.168.204.105@tcp122 [8/256/0/180] [13522.178302] LNet: Added LNI 192.168.204.105@tcp123 [8/256/0/180] [13522.470359] LNet: Added LNI 192.168.204.105@tcp124 [8/256/0/180] [13522.763185] LNet: Added LNI 192.168.204.105@tcp125 [8/256/0/180] [13523.050321] LNet: Added LNI 192.168.204.105@tcp126 [8/256/0/180] [13523.341430] LNet: Added LNI 192.168.204.105@tcp127 [8/256/0/180] [13523.630892] LNet: Added LNI 192.168.204.105@tcp128 [8/256/0/180] [13523.927090] LNet: Added LNI 192.168.204.105@tcp129 [8/256/0/180] [13524.230096] LNet: Added LNI 192.168.204.105@tcp130 [8/256/0/180] [13524.549939] LNet: Added LNI 192.168.204.105@tcp131 [8/256/0/180] [13524.832197] LNet: Added LNI 192.168.204.105@tcp132 [8/256/0/180] [13525.127509] LNet: Added LNI 192.168.204.105@tcp133 [8/256/0/180] [13525.438038] LNet: Added LNI 192.168.204.105@tcp134 [8/256/0/180] [13525.752497] LNet: Added LNI 192.168.204.105@tcp135 [8/256/0/180] [13535.346602] Lustre: DEBUG MARKER: conf-sanity test_73c: @@@@@@ FAIL: unable to start ost1 [13543.798305] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [13544.913926] Key type lgssc unregistered [13545.085266] LNet: 817741:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13545.087773] LNetError: 817741:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [13545.089758] LNet: Removed LNI 192.168.204.105@tcp [13545.091208] LNet: Removed LNI 192.168.204.105@tcp100 [13545.092535] LNet: Removed LNI 192.168.204.105@tcp101 [13545.093630] LNet: Removed LNI 192.168.204.105@tcp102 [13545.094962] LNet: Removed LNI 192.168.204.105@tcp103 [13545.096074] LNet: Removed LNI 192.168.204.105@tcp104 [13545.097121] LNet: Removed LNI 192.168.204.105@tcp105 [13545.098326] LNet: Removed LNI 192.168.204.105@tcp106 [13545.099404] LNet: Removed LNI 192.168.204.105@tcp107 [13545.100660] LNet: Removed LNI 192.168.204.105@tcp108 [13545.101706] LNet: Removed LNI 192.168.204.105@tcp109 [13545.102775] LNet: Removed LNI 192.168.204.105@tcp110 [13545.103823] LNet: Removed LNI 192.168.204.105@tcp111 [13545.104892] LNet: Removed LNI 192.168.204.105@tcp112 [13545.105964] LNet: Removed LNI 192.168.204.105@tcp113 [13545.107059] LNet: Removed LNI 192.168.204.105@tcp114 [13545.108099] LNet: Removed LNI 192.168.204.105@tcp115 [13545.109158] LNet: Removed LNI 192.168.204.105@tcp116 [13545.110206] LNet: Removed LNI 192.168.204.105@tcp117 [13545.111254] LNet: Removed LNI 192.168.204.105@tcp118 [13545.112248] LNet: Removed LNI 192.168.204.105@tcp119 [13545.113211] LNet: Removed LNI 192.168.204.105@tcp120 [13545.114237] LNet: Removed LNI 192.168.204.105@tcp121 [13545.115267] LNet: Removed LNI 192.168.204.105@tcp122 [13545.116395] LNet: Removed LNI 192.168.204.105@tcp123 [13545.117453] LNet: Removed LNI 192.168.204.105@tcp124 [13545.118491] LNet: Removed LNI 192.168.204.105@tcp125 [13545.120218] LNet: Removed LNI 192.168.204.105@tcp126 [13545.122151] LNet: Removed LNI 192.168.204.105@tcp127 [13545.124621] LNet: Removed LNI 192.168.204.105@tcp128 [13545.127188] LNet: Removed LNI 192.168.204.105@tcp129 [13545.130636] LNet: Removed LNI 192.168.204.105@tcp130 [13545.132275] LNet: Removed LNI 192.168.204.105@tcp131 [13545.133991] LNet: Removed LNI 192.168.204.105@tcp132 [13545.135723] LNet: Removed LNI 192.168.204.105@tcp133 [13545.137433] LNet: Removed LNI 192.168.204.105@tcp134 [13545.143470] LNet: Removed LNI 192.168.204.105@tcp135 [13545.425889] Key type .llcrypt unregistered [13545.426889] Key type ._llcrypt unregistered [13554.922623] Key type ._llcrypt registered [13554.923641] Key type .llcrypt registered [13554.962906] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [13578.802863] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [13579.420108] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13579.425616] alg: No test for adler32 (adler32-zlib) [13580.289488] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [13580.375911] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [13581.959173] Key type lgssc registered [13582.393949] Lustre: Echo OBD driver; http://www.lustre.org/ [13588.987615] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13589.415225] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13590.276070] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13591.035311] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [13593.131843] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 23:32:28 (1767328348) [13600.409586] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [13601.517306] Key type lgssc unregistered [13601.650450] LNet: 826846:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13601.654208] LNetError: 826846:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [13601.664432] LNet: Removed LNI 192.168.204.105@tcp [13602.022541] Key type .llcrypt unregistered [13602.024490] Key type ._llcrypt unregistered [13626.655103] Key type ._llcrypt registered [13626.657179] Key type .llcrypt registered [13626.697334] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [13627.129972] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13627.181935] alg: No test for adler32 (adler32-zlib) [13628.053027] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [13628.151870] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [13629.743181] Key type lgssc registered [13630.169571] Lustre: Echo OBD driver; http://www.lustre.org/ [13641.217884] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing lnet_if_list [13642.939875] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [13642.953630] LNet: Added LNI 192.168.204.105@tcp10 [8/256/0/180] [13644.897325] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [13644.912663] LNet: Added LNI 192.168.204.105@tcp11 [8/256/0/180] [13646.271418] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [13646.282872] LNet: Added LNI 192.168.204.105@tcp12 [8/256/0/180] [13647.583967] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [13647.595458] LNet: Added LNI 192.168.204.105@tcp13 [8/256/0/180] [13648.989829] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [13649.002654] LNet: Added LNI 192.168.204.105@tcp14 [8/256/0/180] [13650.557313] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [13650.569017] LNet: Added LNI 192.168.204.105@tcp15 [8/256/0/180] [13652.200893] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [13652.213177] LNet: Added LNI 192.168.204.105@tcp16 [8/256/0/180] [13653.674327] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [13653.685681] LNet: Added LNI 192.168.204.105@tcp17 [8/256/0/180] [13655.231556] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [13655.240571] LNet: Added LNI 192.168.204.105@tcp18 [8/256/0/180] [13656.679536] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [13656.691044] LNet: Added LNI 192.168.204.105@tcp19 [8/256/0/180] [13658.246678] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [13658.258713] LNet: Added LNI 192.168.204.105@tcp20 [8/256/0/180] [13659.789051] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [13659.798091] LNet: Added LNI 192.168.204.105@tcp21 [8/256/0/180] [13661.265784] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [13661.278266] LNet: Added LNI 192.168.204.105@tcp22 [8/256/0/180] [13662.735912] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [13662.748562] LNet: Added LNI 192.168.204.105@tcp23 [8/256/0/180] [13664.386582] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [13664.397175] LNet: Added LNI 192.168.204.105@tcp24 [8/256/0/180] [13666.245342] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [13666.256686] LNet: Added LNI 192.168.204.105@tcp25 [8/256/0/180] [13668.072361] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [13668.083218] LNet: Added LNI 192.168.204.105@tcp26 [8/256/0/180] [13669.978113] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [13669.989081] LNet: Added LNI 192.168.204.105@tcp27 [8/256/0/180] [13671.891781] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [13671.902520] LNet: Added LNI 192.168.204.105@tcp28 [8/256/0/180] [13673.798670] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [13673.808118] LNet: Added LNI 192.168.204.105@tcp29 [8/256/0/180] [13675.621584] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [13675.633875] LNet: Added LNI 192.168.204.105@tcp30 [8/256/0/180] [13677.639529] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [13677.650439] LNet: Added LNI 192.168.204.105@tcp31 [8/256/0/180] [13679.697840] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [13679.711688] LNet: Added LNI 192.168.204.105@tcp32 [8/256/0/180] [13681.775676] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [13681.786593] LNet: Added LNI 192.168.204.105@tcp33 [8/256/0/180] [13683.839178] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [13683.848774] LNet: Added LNI 192.168.204.105@tcp34 [8/256/0/180] [13686.210981] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [13686.223928] LNet: Added LNI 192.168.204.105@tcp35 [8/256/0/180] [13688.645972] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [13688.659897] LNet: Added LNI 192.168.204.105@tcp36 [8/256/0/180] [13691.010297] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [13691.022592] LNet: Added LNI 192.168.204.105@tcp37 [8/256/0/180] [13693.255295] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [13693.267996] LNet: Added LNI 192.168.204.105@tcp38 [8/256/0/180] [13695.663912] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [13695.674545] LNet: Added LNI 192.168.204.105@tcp39 [8/256/0/180] [13698.071659] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [13698.084745] LNet: Added LNI 192.168.204.105@tcp40 [8/256/0/180] [13700.539777] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [13700.553238] LNet: Added LNI 192.168.204.105@tcp41 [8/256/0/180] [13703.068374] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [13703.082542] LNet: Added LNI 192.168.204.105@tcp42 [8/256/0/180] [13705.599265] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [13705.610807] LNet: Added LNI 192.168.204.105@tcp43 [8/256/0/180] [13708.203247] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [13708.216589] LNet: Added LNI 192.168.204.105@tcp44 [8/256/0/180] [13710.816095] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [13710.828773] LNet: Added LNI 192.168.204.105@tcp45 [8/256/0/180] [13713.649215] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [13713.660467] LNet: Added LNI 192.168.204.105@tcp46 [8/256/0/180] [13716.271473] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [13716.286553] LNet: Added LNI 192.168.204.105@tcp47 [8/256/0/180] [13718.901623] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [13718.912531] LNet: Added LNI 192.168.204.105@tcp48 [8/256/0/180] [13721.703852] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [13721.715080] LNet: Added LNI 192.168.204.105@tcp49 [8/256/0/180] [13724.553729] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [13724.564392] LNet: Added LNI 192.168.204.105@tcp50 [8/256/0/180] [13893.105511] Lustre: DEBUG MARKER: conf-sanity test_73e: @@@@@@ FAIL: Mount fails on 192.168.204.105@tcp50 [13901.273280] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [13902.354382] Key type lgssc unregistered [13902.513368] LNet: 854873:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13902.517055] LNetError: 854873:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [13902.520321] LNet: Removed LNI 192.168.204.105@tcp [13902.521893] LNet: Removed LNI 192.168.204.105@tcp10 [13902.523596] LNet: Removed LNI 192.168.204.105@tcp11 [13902.525253] LNet: Removed LNI 192.168.204.105@tcp12 [13902.526932] LNet: Removed LNI 192.168.204.105@tcp13 [13902.528539] LNet: Removed LNI 192.168.204.105@tcp14 [13902.530190] LNet: Removed LNI 192.168.204.105@tcp15 [13902.531811] LNet: Removed LNI 192.168.204.105@tcp16 [13902.533503] LNet: Removed LNI 192.168.204.105@tcp17 [13902.535171] LNet: Removed LNI 192.168.204.105@tcp18 [13902.536760] LNet: Removed LNI 192.168.204.105@tcp19 [13902.538434] LNet: Removed LNI 192.168.204.105@tcp20 [13902.540065] LNet: Removed LNI 192.168.204.105@tcp21 [13902.541815] LNet: Removed LNI 192.168.204.105@tcp22 [13902.543380] LNet: Removed LNI 192.168.204.105@tcp23 [13902.545074] LNet: Removed LNI 192.168.204.105@tcp24 [13902.547697] LNet: Removed LNI 192.168.204.105@tcp25 [13902.549309] LNet: Removed LNI 192.168.204.105@tcp26 [13902.550933] LNet: Removed LNI 192.168.204.105@tcp27 [13902.552498] LNet: Removed LNI 192.168.204.105@tcp28 [13902.554115] LNet: Removed LNI 192.168.204.105@tcp29 [13902.555925] LNet: Removed LNI 192.168.204.105@tcp30 [13902.557540] LNet: Removed LNI 192.168.204.105@tcp31 [13902.559183] LNet: Removed LNI 192.168.204.105@tcp32 [13902.560930] LNet: Removed LNI 192.168.204.105@tcp33 [13902.562542] LNet: Removed LNI 192.168.204.105@tcp34 [13902.564183] LNet: Removed LNI 192.168.204.105@tcp35 [13902.565815] LNet: Removed LNI 192.168.204.105@tcp36 [13902.567396] LNet: Removed LNI 192.168.204.105@tcp37 [13902.569100] LNet: Removed LNI 192.168.204.105@tcp38 [13902.570725] LNet: Removed LNI 192.168.204.105@tcp39 [13902.572397] LNet: Removed LNI 192.168.204.105@tcp40 [13902.574048] LNet: Removed LNI 192.168.204.105@tcp41 [13902.575520] LNet: Removed LNI 192.168.204.105@tcp42 [13902.577069] LNet: Removed LNI 192.168.204.105@tcp43 [13902.578594] LNet: Removed LNI 192.168.204.105@tcp44 [13902.580154] LNet: Removed LNI 192.168.204.105@tcp45 [13902.581568] LNet: Removed LNI 192.168.204.105@tcp46 [13902.583107] LNet: Removed LNI 192.168.204.105@tcp47 [13902.584609] LNet: Removed LNI 192.168.204.105@tcp48 [13902.586097] LNet: Removed LNI 192.168.204.105@tcp49 [13902.591420] LNet: Removed LNI 192.168.204.105@tcp50 [13902.881462] Key type .llcrypt unregistered [13902.882693] Key type ._llcrypt unregistered [13928.659832] Key type ._llcrypt registered [13928.660941] Key type .llcrypt registered [13928.699536] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [13929.021496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13929.050099] alg: No test for adler32 (adler32-zlib) [13929.900189] Lustre: Lustre: Build Version: 2.17.0_RC4_4_g6421352 [13929.989921] LNet: Added LNI 192.168.204.105@tcp [8/256/0/180] [13931.567140] Key type lgssc registered [13931.927549] Lustre: Echo OBD driver; http://www.lustre.org/ [13935.266775] Lustre: DEBUG MARKER: conf-sanity test_74: @@@@@@ FAIL: MDT start failed [13936.952207] Lustre: DEBUG MARKER: == conf-sanity test 75: The order of --index should be irrelevant ========================================================== 23:38:11 (1767328691) [13962.970034] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [13967.300138] Lustre: DEBUG MARKER: conf-sanity test_75: @@@@@@ FAIL: add mds1 failed for default params [13976.661541] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_hostid [13998.803583] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing load_modules_local [14005.567614] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14006.136161] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14006.736537] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14007.449898] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [14007.996981] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13612 sec ========= 23:39:22 (1767328762) [14008.534530] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 23:39:23 (1767328763) === [14009.629691] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 23:39:24 (1767328764) === [14018.104494] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing unload_modules_local [14019.141977] Key type lgssc unregistered [14019.270307] LNet: 872826:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14019.272483] LNetError: 872826:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [14019.279322] LNet: Removed LNI 192.168.204.105@tcp [14019.578475] Key type .llcrypt unregistered [14019.579465] Key type ._llcrypt unregistered