[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 491709352 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002340] x2apic enabled [ 0.003010] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.006658] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008012] pid_max: default: 32768 minimum: 301 [ 0.009134] LSM: Security Framework initializing [ 0.010051] Yama: becoming mindful. [ 0.011049] SELinux: Initializing. [ 0.012072] *** VALIDATE selinux *** [ 0.020386] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025395] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026167] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027127] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028126] *** VALIDATE tmpfs *** [ 0.030200] *** VALIDATE proc *** [ 0.031199] *** VALIDATE cgroup *** [ 0.031902] *** VALIDATE cgroup2 *** [ 0.032273] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033166] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035005] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036031] Spectre V2 : User space: Vulnerable [ 0.037008] Speculative Store Bypass: Vulnerable [ 0.040519] debug: unmapping init [mem 0xffffffffb5e59000-0xffffffffb5e60fff] [ 0.042202] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043741] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044026] ... version: 2 [ 0.045024] ... bit width: 48 [ 0.046015] ... generic registers: 4 [ 0.047014] ... value mask: 0000ffffffffffff [ 0.048014] ... max period: 00007fffffffffff [ 0.049015] ... fixed-purpose events: 3 [ 0.050017] ... event mask: 000000070000000f [ 0.051320] rcu: Hierarchical SRCU implementation. [ 0.053524] smp: Bringing up secondary CPUs ... [ 0.054683] x86: Booting SMP configuration: [ 0.055039] .... node #0, CPUs: #1 #2 #3 [ 0.058519] smp: Brought up 1 node, 4 CPUs [ 0.060013] smpboot: Max logical packages: 1 [ 0.061018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.133599] node 0 deferred pages initialised in 70ms [ 0.137118] devtmpfs: initialized [ 0.138338] x86/mm: Memory block size: 128MB [ 0.142437] gcov: version magic: 0x41383552 [ 0.146289] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.147087] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.148295] pinctrl core: initialized pinctrl subsystem [ 0.149217] [ 0.150012] ************************************************************* [ 0.151013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.152016] ** ** [ 0.153016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.154014] ** ** [ 0.155011] ** This means that this kernel is built to expose internal ** [ 0.156016] ** IOMMU data structures, which may compromise security on ** [ 0.157016] ** your system. ** [ 0.158014] ** ** [ 0.159012] ** If you see this message and you are not debugging the ** [ 0.160013] ** kernel, report this immediately to your vendor! ** [ 0.161014] ** ** [ 0.162014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.163011] ************************************************************* [ 0.164666] NET: Registered protocol family 16 [ 0.165447] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.166072] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.167060] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.168450] cpuidle: using governor menu [ 0.170242] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.172465] PCI: Using configuration type 1 for base access [ 0.175133] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.184121] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.185043] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.187088] cryptd: max_cpu_qlen set to 1000 [ 0.189273] ACPI: Added _OSI(Module Device) [ 0.190013] ACPI: Added _OSI(Processor Device) [ 0.191012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.192014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.196093] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.198588] ACPI: Interpreter enabled [ 0.199063] ACPI: PM: (supports S0 S3 S4 S5) [ 0.200014] ACPI: Using IOAPIC for interrupt routing [ 0.201129] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.202422] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.212149] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.213059] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.214032] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.215086] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.217464] acpiphp: Slot [2] registered [ 0.218148] acpiphp: Slot [5] registered [ 0.219169] acpiphp: Slot [6] registered [ 0.220140] acpiphp: Slot [7] registered [ 0.221113] acpiphp: Slot [8] registered [ 0.222142] acpiphp: Slot [9] registered [ 0.223096] acpiphp: Slot [10] registered [ 0.224136] acpiphp: Slot [3] registered [ 0.225148] acpiphp: Slot [4] registered [ 0.226121] acpiphp: Slot [11] registered [ 0.227114] acpiphp: Slot [12] registered [ 0.228098] acpiphp: Slot [13] registered [ 0.229136] acpiphp: Slot [14] registered [ 0.231137] acpiphp: Slot [15] registered [ 0.233124] acpiphp: Slot [16] registered [ 0.235115] acpiphp: Slot [17] registered [ 0.237141] acpiphp: Slot [18] registered [ 0.239122] acpiphp: Slot [19] registered [ 0.241105] acpiphp: Slot [20] registered [ 0.242110] acpiphp: Slot [21] registered [ 0.244190] acpiphp: Slot [22] registered [ 0.246171] acpiphp: Slot [23] registered [ 0.247130] acpiphp: Slot [24] registered [ 0.249124] acpiphp: Slot [25] registered [ 0.250124] acpiphp: Slot [26] registered [ 0.252108] acpiphp: Slot [27] registered [ 0.253376] acpiphp: Slot [28] registered [ 0.255090] acpiphp: Slot [29] registered [ 0.256109] acpiphp: Slot [30] registered [ 0.257189] acpiphp: Slot [31] registered [ 0.259076] PCI host bridge to bus 0000:00 [ 0.260016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.262027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.265024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.267031] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.270029] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.273048] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.275263] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.279286] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.283615] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.295836] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.301105] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.304022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.306020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.310024] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.313483] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.316723] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.320043] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.323935] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.330018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.343899] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.347921] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.354621] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.362020] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.371016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.398023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.409000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.417112] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.424020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.439017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.449622] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.459017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.473016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.499022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.509413] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.516015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.522013] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.536017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.550352] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.558015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.568024] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.587019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.597642] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.605016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.617017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.639037] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.651000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.654418] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.656331] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.658406] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.660318] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.665028] iommu: Default domain type: Passthrough [ 0.667456] SCSI subsystem initialized [ 0.668146] ACPI: bus type USB registered [ 0.670143] usbcore: registered new interface driver usbfs [ 0.671076] usbcore: registered new interface driver hub [ 0.673096] usbcore: registered new device driver usb [ 0.674177] pps_core: LinuxPPS API ver. 1 registered [ 0.676013] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.678100] PTP clock support registered [ 0.680155] EDAC MC: Ver: 3.0.0 [ 0.683113] PCI: Using ACPI for IRQ routing [ 0.685945] NetLabel: Initializing [ 0.687011] NetLabel: domain hash size = 128 [ 0.688012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.690092] NetLabel: unlabeled traffic allowed by default [ 0.693166] vgaarb: loaded [ 0.695007] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.696012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.701602] clocksource: Switched to clocksource kvm-clock [ 0.816317] VFS: Disk quotas dquot_6.6.0 [ 0.818987] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.821567] *** VALIDATE ramfs *** [ 0.822571] *** VALIDATE hugetlbfs *** [ 0.824238] pnp: PnP ACPI init [ 0.826294] pnp: PnP ACPI: found 6 devices [ 0.852456] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.855245] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.857531] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.859497] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.861556] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.864184] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.866544] NET: Registered protocol family 2 [ 0.869837] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.875872] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.880935] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.885440] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.888392] TCP: Hash tables configured (established 65536 bind 65536) [ 0.890611] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.893021] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.895275] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.897552] NET: Registered protocol family 1 [ 0.899811] RPC: Registered named UNIX socket transport module. [ 0.901576] RPC: Registered udp transport module. [ 0.903043] RPC: Registered tcp transport module. [ 0.904285] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.906189] NET: Registered protocol family 44 [ 0.907421] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.908934] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.910303] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.911840] PCI: CLS 0 bytes, default 64 [ 0.913845] Unpacking initramfs... [ 2.344276] debug: unmapping init [mem 0xffff9d70fcc54000-0xffff9d70fffbffff] [ 2.348085] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.350525] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.353745] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.848143] Initialise system trusted keyrings [ 2.850160] Key type blacklist registered [ 2.852699] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.861861] zbud: loaded [ 2.864807] *** VALIDATE nfs *** [ 2.866104] *** VALIDATE nfs4 *** [ 2.867785] pstore: using deflate compression [ 2.871936] Platform Keyring initialized [ 2.981830] NET: Registered protocol family 38 [ 2.983705] Key type asymmetric registered [ 2.985289] Asymmetric key parser 'x509' registered [ 2.987292] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.990129] io scheduler mq-deadline registered [ 2.991962] io scheduler kyber registered [ 2.993678] io scheduler bfq registered [ 2.995656] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.999126] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.002383] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.005669] ACPI: Power Button [PWRF] [ 3.010880] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.018861] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.040128] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.054172] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.077047] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.104104] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.132639] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.139171] Non-volatile memory driver v1.3 [ 3.141088] Linux agpgart interface v0.103 [ 3.175952] virtio_blk virtio1: [vda] 146624 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.179057] vda: detected capacity change from 0 to 75071488 [ 3.199168] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.203509] vdb: detected capacity change from 0 to 1073741824 [ 3.223256] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.226723] vdc: detected capacity change from 0 to 2621440000 [ 3.247084] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.251408] vdd: detected capacity change from 0 to 2621440000 [ 3.268449] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.271285] vde: detected capacity change from 0 to 4294967296 [ 3.288083] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.291392] vdf: detected capacity change from 0 to 4294967296 [ 3.301845] libphy: Fixed MDIO Bus: probed [ 3.310929] usbcore: registered new interface driver usbserial_generic [ 3.313714] usbserial: USB Serial support registered for generic [ 3.316321] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.320747] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.322655] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.325161] mousedev: PS/2 mouse device common for all mice [ 3.328307] rtc_cmos 00:05: RTC can wake from S4 [ 3.331562] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.331804] rtc_cmos 00:05: registered as rtc0 [ 3.340429] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.344155] intel_pstate: CPU model not supported [ 3.347531] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.351679] hid: raw HID events driver (C) Jiri Kosina [ 3.354524] usbcore: registered new interface driver usbhid [ 3.354753] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.356322] usbhid: USB HID core driver [ 3.360562] drop_monitor: Initializing network drop monitor service [ 3.363335] Initializing XFRM netlink socket [ 3.365513] NET: Registered protocol family 10 [ 3.368729] Segment Routing with IPv6 [ 3.370229] NET: Registered protocol family 17 [ 3.371953] mpls_gso: MPLS GSO support [ 3.378750] RAS: Correctable Errors collector initialized. [ 3.381729] AVX version of gcm_enc/dec engaged. [ 3.383379] AES CTR mode by8 optimization enabled [ 3.459809] sched_clock: Marking stable (3459785814, 0)->(4428243664, -968457850) [ 3.463974] registered taskstats version 1 [ 3.466705] Loading compiled-in X.509 certificates [ 3.469144] zswap: loaded using pool lzo/zbud [ 3.495391] Key type big_key registered [ 3.507821] Key type encrypted registered [ 3.509781] ima: No TPM chip found, activating TPM-bypass! [ 3.511396] ima: Allocated hash algorithm: sha1 [ 3.513838] ima: No architecture policies found [ 3.515049] evm: Initialising EVM extended attributes: [ 3.516214] evm: security.selinux [ 3.516887] evm: security.ima [ 3.517504] evm: security.capability [ 3.518307] evm: HMAC attrs: 0x1 [ 3.520110] rtc_cmos 00:05: setting system clock to 2026-08-27 11:35:29 UTC (1787830529) [ 3.525217] debug: unmapping init [mem 0xffffffffb6e03000-0xffffffffb6ffffff] [ 3.527775] debug: unmapping init [mem 0xffffffffb5b82000-0xffffffffb5e58fff] [ 3.535222] Write protecting the kernel read-only data: 28672k [ 3.538614] debug: unmapping init [mem 0xffffffffb4203000-0xffffffffb43fffff] [ 3.541623] debug: unmapping init [mem 0xffffffffb4b14000-0xffffffffb4bfffff] [ 3.574083] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.581340] systemd[1]: Detected virtualization kvm. [ 3.582751] systemd[1]: Detected architecture x86-64. [ 3.584158] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.609327] systemd[1]: No hostname configured. [ 3.610706] systemd[1]: Set hostname to . [ 3.612300] random: systemd: uninitialized urandom read (16 bytes read) [ 3.614549] systemd[1]: Initializing machine ID from random generator. [ 3.645963] random: ln: uninitialized urandom read (6 bytes read) [ 3.737498] random: systemd: uninitialized urandom read (16 bytes read) [ 3.740572] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 3.745595] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.752880] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ OK ] Reached target Slices. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... [ OK ] Reached target Sockets. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Apply Kernel Variables. [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.333551] device-mapper: uevent: version 1.0.3 [ 4.335581] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.026568] virtio_net virtio0 ens2: renamed from eth0 [ 5.043492] random: fast init done [ 5.131722] scsi host0: ata_piix [ 5.178201] scsi host1: ata_piix [ 5.180256] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.183666] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.263954] dracut-initqueue[589]: RTNETLINK answers: File exists [ 10.027314] random: crng init done [ 10.028479] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.302133] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.552623] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.824273] SELinux: Disabled at runtime. [ 11.883661] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.894563] systemd[1]: Detected virtualization kvm. [ 11.896545] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.377293] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.380927] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.385840] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.390638] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.393917] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.401249] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.412228] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ 12.431760] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Reached target RPC Port Mapper. Mounting POSIX Message Queue File System... Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ 12.810398] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.129573] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.156632] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.236348] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.248328] EDAC sbridge: Ver: 1.1.2 [ 17.824795] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (6s / no limit)[ 18.688569] NFS: Registering the id_resolver key type [ 18.696965] Key type id_resolver registered [ 18.700833] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (6s / no limit) [*** ] A start job is running for Configur…-only root support (6s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg420-server login: [ 87.655977] libcfs: loading out-of-tree module taints kernel. [ 87.690794] Key type ._llcrypt registered [ 87.693093] Key type .llcrypt registered [ 87.782344] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 113.088568] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 116.089095] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 116.129506] alg: No test for adler32 (adler32-zlib) [ 117.770604] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 119.264196] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 121.176466] Key type lgssc registered [ 123.730487] Lustre: Echo OBD driver; http://www.lustre.org/ [ 131.999253] hrtimer: interrupt took 3239522 ns [ 142.032636] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 188.171782] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 203.623530] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 203.650906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 204.978968] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 205.042736] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 205.109746] Lustre: lustre-MDT0000: new disk, initializing [ 205.166524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 205.197394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 210.567503] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 226.950549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 227.102910] Lustre: 6521:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 227.148424] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 227.156097] Lustre: Skipped 1 previous similar message [ 227.344266] Lustre: lustre-MDT0001: new disk, initializing [ 227.479145] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 227.528403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 227.547708] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 232.660863] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 238.125359] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 248.997375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 249.282497] Lustre: lustre-OST0000: new disk, initializing [ 249.290435] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 249.297502] Lustre: 8458:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 249.351433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 255.129463] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 258.095988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 258.104183] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 258.196697] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 271.646363] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 271.844479] Lustre: lustre-OST0001: new disk, initializing [ 271.856499] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 271.871732] Lustre: 9532:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 271.944233] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 279.155961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 279.167531] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 279.263946] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 279.787383] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 293.639563] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 300.079249] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 307.639556] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing check_logdir /tmp/testlogs/ [ 313.206335] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing yml_node [ 317.547351] Lustre: DEBUG MARKER: Client: 2.17.57.61 [ 320.568369] Lustre: DEBUG MARKER: MDS: 2.17.57.61 [ 323.715555] Lustre: DEBUG MARKER: OSS: 2.17.57.61 [ 325.516926] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 07:40:49 EDT 2026 [ 345.459318] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 347.166397] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 365.537471] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 365.550774] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 365.579197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 366.561837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 366.569266] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 366.576018] Lustre: Skipped 2 previous similar messages [ 368.632778] Lustre: server umount lustre-MDT0000 complete [ 373.259892] LustreError: 8457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787830899 with bad export cookie 9255004811449978473 [ 373.267951] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 373.288472] LustreError: 8457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 376.801103] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 376.803697] LustreError: 8466:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 376.803898] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 376.816823] Lustre: Skipped 3 previous similar messages [ 376.836448] Lustre: Skipped 1 previous similar message [ 376.895951] LustreError: 8466:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 378.852354] LustreError: 6527:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 378.852463] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 378.878225] Lustre: Skipped 2 previous similar messages [ 379.910419] Lustre: server umount lustre-MDT0001 complete [ 384.507368] Lustre: server umount lustre-OST0000 complete [ 388.715158] Lustre: server umount lustre-OST0001 complete [ 395.237720] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 407.940274] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 445.686439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 445.949930] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 445.989605] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 446.074876] Lustre: lustre-MDT0000: new disk, initializing [ 446.170638] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 446.197380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 450.148254] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 460.396772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 460.567086] Lustre: 15084:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 460.662164] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 460.666927] Lustre: Skipped 1 previous similar message [ 460.715773] Lustre: lustre-MDT0001: new disk, initializing [ 460.809481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 460.865488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 460.874078] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 464.883519] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 475.295703] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 482.826677] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 490.432266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 490.820904] Lustre: lustre-OST0000: new disk, initializing [ 490.824169] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 490.834967] Lustre: 17052:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 490.962372] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 492.335825] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 492.352682] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 492.552079] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 497.365617] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 506.463562] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 511.288724] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 511.470402] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 518.446431] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 518.759196] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 523.240405] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 523.246044] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 523.247332] LustreError: Skipped 1 previous similar message [ 523.251100] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 523.293363] Lustre: Skipped 1 previous similar message [ 527.412166] Lustre: server umount lustre-OST0000 complete [ 532.449641] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 532.459173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 532.480657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 532.489713] Lustre: Skipped 1 previous similar message [ 537.965411] Lustre: server umount lustre-MDT0000 complete [ 538.593961] LustreError: 15093:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 538.647262] LustreError: 15093:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 542.591618] LustreError: 15074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831068 with bad export cookie 9255004811449980006 [ 542.621582] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 543.088174] Lustre: server umount lustre-MDT0001 complete [ 547.793752] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 07:44:31 (1787831071) [ 550.357125] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 552.507093] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 07:44:36 (1787831076) [ 559.512329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 559.943915] LustreError: 19460:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 560.012957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 564.308451] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 565.220870] LustreError: 19461:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 572.792066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 577.563799] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 584.686731] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 590.607352] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 597.214637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 604.093969] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 611.408419] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 619.492394] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 619.500671] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 619.515586] Lustre: Skipped 1 previous similar message [ 619.523801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 619.527940] Lustre: Skipped 2 previous similar messages [ 625.477307] Lustre: server umount lustre-OST0000 complete [ 629.729493] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 629.741307] LustreError: Skipped 1 previous similar message [ 635.873178] LustreError: 19461:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 635.888492] LustreError: 19461:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 635.910210] Lustre: server umount lustre-MDT0000 complete [ 639.203914] LustreError: 20196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831165 with bad export cookie 9255004811449980888 [ 639.213933] LustreError: 20196:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 639.218885] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 639.582069] Lustre: server umount lustre-MDT0001 complete [ 650.416970] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 653.802544] Key type lgssc unregistered [ 654.097379] LNet: 22995:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 654.108386] LNetError: 22995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 654.139375] LNet: Removed LNI 192.168.204.120@tcp [ 655.374265] Key type .llcrypt unregistered [ 655.375808] Key type ._llcrypt unregistered [ 680.719077] Key type ._llcrypt registered [ 680.720571] Key type .llcrypt registered [ 680.934863] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 682.293315] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 682.399479] alg: No test for adler32 (adler32-zlib) [ 683.588601] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 683.794621] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 685.481387] Key type lgssc registered [ 686.980557] Lustre: Echo OBD driver; http://www.lustre.org/ [ 696.765670] loop: module loaded [ 696.841798] loop0: detected capacity change from 0 to 204800000 [ 697.032521] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 697.045033] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 697.518782] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 703.429759] loop0: detected capacity change from 0 to 204800000 [ 703.600961] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 703.621951] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 703.636035] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 709.060272] loop0: detected capacity change from 0 to 204800000 [ 714.686705] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 714.727571] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 716.051512] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 716.105591] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 716.236506] Lustre: lustre-MDT0000: new disk, initializing [ 716.319859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 716.333410] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 721.443986] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 731.158784] loop1: detected capacity change from 0 to 204800000 [ 737.163390] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 737.381485] Lustre: lustre-OST07c6: new disk, initializing [ 737.386468] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 737.389399] Lustre: Skipped 1 previous similar message [ 737.395827] Lustre: 26188:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 737.468474] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 742.958350] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 742.964420] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 742.995223] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 743.042602] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 747.829573] Lustre: 25172:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 747.846938] Lustre: 25172:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 747.854293] Lustre: 25172:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 747.865532] Lustre: 25172:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 747.876350] Lustre: 25172:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 747.882487] Lustre: 25172:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 763.060758] Lustre: 25173:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 763.076655] Lustre: 25173:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 763.091769] Lustre: 25173:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 763.097861] Lustre: 25173:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 763.110286] Lustre: 25173:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 763.115866] Lustre: 25173:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 768.166297] Lustre: 26132:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 768.183600] Lustre: 26132:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 768.195195] Lustre: 26132:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 768.199962] Lustre: 26132:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 768.207309] Lustre: 26132:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 768.214124] Lustre: 26132:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 768.224185] Lustre: 26132:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 768.234550] Lustre: 26132:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 768.246987] Lustre: 26132:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 768.255881] Lustre: 26132:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 768.266800] Lustre: 26132:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 768.275271] Lustre: 26132:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 771.555219] Lustre: 25174:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 771.575481] Lustre: 25174:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 771.589306] Lustre: 25174:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 771.601715] Lustre: 25174:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 771.626095] Lustre: 25174:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 771.634123] Lustre: 25174:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 771.641847] Lustre: 25174:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 771.651330] Lustre: 25174:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 771.660394] Lustre: 25174:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 771.671656] Lustre: 25174:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 771.679462] Lustre: 25174:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 771.695624] Lustre: 25174:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 777.149393] Lustre: 25172:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 777.168299] Lustre: 25172:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 777.187493] Lustre: 25172:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 777.211567] Lustre: 25172:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 777.224609] Lustre: 25172:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 777.236826] Lustre: 25172:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 777.246361] Lustre: 25172:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 777.251651] Lustre: 25172:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 777.256809] Lustre: 25172:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 777.262130] Lustre: 25172:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 777.267175] Lustre: 25172:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 777.274407] Lustre: 25172:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 785.519869] Lustre: 25172:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 785.548659] Lustre: 25172:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 785.565149] Lustre: 25172:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 785.589812] Lustre: 25172:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 785.616844] Lustre: 25172:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 785.630076] Lustre: 25172:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 785.638721] Lustre: 25172:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 785.646101] Lustre: 25172:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 785.673131] Lustre: 25172:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 785.682888] Lustre: 25172:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 785.690633] Lustre: 25172:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 785.698491] Lustre: 25172:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 840.161885] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 840.167476] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 840.208529] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 843.195951] Lustre: server umount lustre-OST07c6 complete [ 854.182876] Lustre: server umount lustre-MDT0000 complete [ 875.224447] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 878.750786] Key type lgssc unregistered [ 879.105956] LNet: 28744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 879.118291] LNetError: 28744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 879.135192] LNet: Removed LNI 192.168.204.120@tcp [ 880.001169] Key type .llcrypt unregistered [ 880.003933] Key type ._llcrypt unregistered [ 901.729752] Key type ._llcrypt registered [ 901.733407] Key type .llcrypt registered [ 901.943694] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 903.387315] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 903.416064] alg: No test for adler32 (adler32-zlib) [ 904.551727] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 904.861368] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 906.583869] Key type lgssc registered [ 907.586224] Lustre: Echo OBD driver; http://www.lustre.org/ [ 915.949358] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 915.976168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 917.600621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 922.594204] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 927.718037] LustreError: 30031:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 927.751381] LustreError: 30031:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 932.624483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 932.832854] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 932.999938] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 937.776747] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 946.317137] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 953.150238] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 960.712978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 961.147853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 962.199583] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 968.151861] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 976.010518] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 993.249389] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 993.256792] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 993.267687] LustreError: Skipped 1 previous similar message [ 993.300839] Lustre: Skipped 1 previous similar message [ 993.317810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 993.325066] Lustre: Skipped 1 previous similar message [ 994.514362] Lustre: server umount lustre-OST0000 complete [ 999.392442] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 999.406176] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 999.418874] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1003.493255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1003.494837] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1008.611336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1008.620843] Lustre: Skipped 2 previous similar messages [ 1013.216570] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1013.555264] Lustre: server umount lustre-MDT0000 complete [ 1013.728381] LustreError: 31875:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1013.792061] LustreError: 31875:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1017.650167] LustreError: 30013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831543 with bad export cookie 9885663983739605737 [ 1017.666320] LustreError: 30013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1017.668102] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1018.061203] Lustre: server umount lustre-MDT0001 complete [ 1033.026247] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 1036.143755] Key type lgssc unregistered [ 1036.459800] LNet: 33776:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1036.476909] LNetError: 33776:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1036.496244] LNet: Removed LNI 192.168.204.120@tcp [ 1037.587752] Key type .llcrypt unregistered [ 1037.592315] Key type ._llcrypt unregistered [ 1060.683934] Key type ._llcrypt registered [ 1060.688148] Key type .llcrypt registered [ 1060.859902] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 1062.592377] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1062.711304] alg: No test for adler32 (adler32-zlib) [ 1063.882677] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 1064.298277] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 1066.048910] Key type lgssc registered [ 1067.272762] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1077.233881] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1077.248768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1078.717300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1083.143546] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1088.995889] LustreError: 35052:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1089.056030] LustreError: 35052:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1093.461390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1094.068387] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1098.273320] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1109.358941] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1116.339527] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1123.342627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1123.825288] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1124.872704] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1132.467430] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1142.483295] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1158.624477] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1158.630036] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1158.640778] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1158.669514] Lustre: Skipped 1 previous similar message [ 1158.684837] Lustre: Skipped 1 previous similar message [ 1163.078565] Lustre: server umount lustre-OST0000 complete [ 1168.878677] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1168.901958] Lustre: Skipped 1 previous similar message [ 1168.918934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1168.930992] Lustre: Skipped 1 previous similar message [ 1173.840247] Lustre: server umount lustre-MDT0000 complete [ 1173.985371] LustreError: 35780:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1174.004271] LustreError: 35780:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1178.090101] LustreError: 35047:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1178.109742] LustreError: 35047:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1178.179111] LustreError: 35032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831704 with bad export cookie 5600510682408717373 [ 1178.197178] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1178.673062] Lustre: server umount lustre-MDT0001 complete [ 1191.928851] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 1195.207725] Key type lgssc unregistered [ 1195.467118] LNet: 38608:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1195.473891] LNetError: 38608:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1195.503381] LNet: Removed LNI 192.168.204.120@tcp [ 1196.328922] Key type .llcrypt unregistered [ 1196.335972] Key type ._llcrypt unregistered [ 1223.733440] Key type ._llcrypt registered [ 1223.740954] Key type .llcrypt registered [ 1223.871678] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 1225.148763] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1225.158744] alg: No test for adler32 (adler32-zlib) [ 1226.222193] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 1226.550608] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 1228.248247] Key type lgssc registered [ 1229.520518] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1236.964892] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1237.010249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1238.573141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1243.634960] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1248.769474] LustreError: 40026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1248.788188] LustreError: 40026:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1253.858785] LustreError: 40025:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1254.871104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1255.365342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1260.151926] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1269.167790] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1276.262422] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1284.555195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1284.969726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1292.437190] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1301.087227] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1308.534099] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1318.889982] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1318.895681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1318.923819] Lustre: Skipped 2 previous similar messages [ 1318.935426] Lustre: Skipped 1 previous similar message [ 1322.391325] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.20@tcp (stopping) [ 1322.401829] Lustre: Skipped 1 previous similar message [ 1323.945564] Lustre: server umount lustre-MDT0000 complete [ 1324.004816] LustreError: 40025:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1324.017112] LustreError: 40025:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1327.015494] LustreError: 40753:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1327.867332] LustreError: 40006:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831853 with bad export cookie 2834093244905346574 [ 1327.879843] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1332.197758] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1332.199362] LustreError: 40021:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1332.216177] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1332.235396] LustreError: 40021:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1334.370775] Lustre: server umount lustre-MDT0001 complete [ 1355.485829] Lustre: server umount lustre-OST0000 complete [ 1374.779466] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 1377.680393] Key type lgssc unregistered [ 1377.930748] LNet: 43916:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1377.940881] LNetError: 43916:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1377.974510] LNet: Removed LNI 192.168.204.120@tcp [ 1379.444902] Key type .llcrypt unregistered [ 1379.448820] Key type ._llcrypt unregistered [ 1406.464068] Key type ._llcrypt registered [ 1406.470856] Key type .llcrypt registered [ 1406.581770] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 1408.033306] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1408.061974] alg: No test for adler32 (adler32-zlib) [ 1409.155410] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 1409.388260] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 1411.215260] Key type lgssc registered [ 1412.819837] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1422.634542] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1422.683660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1424.270898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1430.270461] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1434.596639] LustreError: 45331:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1434.615586] LustreError: 45331:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1439.718039] LustreError: 45332:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1440.524786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1440.911153] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1446.932146] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1455.922280] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1463.024924] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1471.086245] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1471.536085] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1478.532651] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1487.387208] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1494.071390] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1499.112516] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1499.116850] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1499.123559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1499.132687] Lustre: Skipped 1 previous similar message [ 1499.140865] Lustre: Skipped 1 previous similar message [ 1504.228479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1504.242066] Lustre: Skipped 2 previous similar messages [ 1504.913040] Lustre: server umount lustre-OST0000 complete [ 1523.311053] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1523.317484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1523.322891] Lustre: Skipped 1 previous similar message [ 1523.333990] Lustre: Skipped 2 previous similar messages [ 1528.801972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1528.811610] Lustre: Skipped 1 previous similar message [ 1529.730108] Lustre: server umount lustre-MDT0000 complete [ 1529.827119] LustreError: 46060:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1529.860558] LustreError: 46060:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1534.554809] LustreError: 45314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832060 with bad export cookie 12985568622391594789 [ 1534.565741] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1534.569474] LustreError: 45314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1541.059209] Lustre: server umount lustre-MDT0001 complete [ 1553.706225] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 1557.004764] Key type lgssc unregistered [ 1557.403834] LNet: 49054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1557.419766] LNetError: 49054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1557.447353] LNet: Removed LNI 192.168.204.120@tcp [ 1558.454539] Key type .llcrypt unregistered [ 1558.455921] Key type ._llcrypt unregistered [ 1590.054579] Key type ._llcrypt registered [ 1590.058210] Key type .llcrypt registered [ 1590.161110] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 1591.604676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1591.646106] alg: No test for adler32 (adler32-zlib) [ 1592.862205] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 1593.272120] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 1594.991523] Key type lgssc registered [ 1596.279761] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1605.223287] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1605.239705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1606.803250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1611.989897] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1616.372948] LustreError: 50471:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1616.397882] LustreError: 50471:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1621.469418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1621.474171] LustreError: 50472:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1621.798472] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1625.627781] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1635.245574] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1642.746269] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1651.820034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1652.273349] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1660.906697] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1670.947255] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1676.855187] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1682.971910] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1688.369957] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1700.742394] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-27 8h03m44s [ 1702.800452] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1707.490758] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1707.498939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1707.510930] Lustre: Skipped 2 previous similar messages [ 1707.533592] Lustre: Skipped 2 previous similar messages [ 1708.583820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1708.596852] Lustre: Skipped 3 previous similar messages [ 1714.684427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1714.691762] Lustre: Skipped 2 previous similar messages [ 1720.291307] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1720.508848] Lustre: server umount lustre-MDT0000 complete [ 1720.813763] LustreError: 50471:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1720.851331] LustreError: 50471:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1724.537658] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1726.948241] LustreError: 50472:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1726.976642] LustreError: 50472:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1732.972062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1733.097496] LustreError: 50471:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1733.143406] LustreError: 50471:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1733.170238] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1733.553524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1733.672161] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1739.280959] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1739.762777] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1743.599476] Lustre: DEBUG MARKER: Wait for df (36722) ... [ 1746.106071] Lustre: DEBUG MARKER: done [ 1753.573909] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1753.584048] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1753.610650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1753.619214] Lustre: Skipped 1 previous similar message [ 1754.083888] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1757.709567] Lustre: server umount lustre-OST0000 complete [ 1765.344411] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1765.349115] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1765.360333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1765.360342] Lustre: Skipped 3 previous similar messages [ 1765.385522] Lustre: Skipped 2 previous similar messages [ 1768.811852] Lustre: server umount lustre-MDT0000 complete [ 1772.516530] LustreError: 51533:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1772.536076] LustreError: 51533:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1772.828667] LustreError: 50451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832298 with bad export cookie 9623141683269925339 [ 1772.841698] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1772.846991] LustreError: 50451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1773.354051] Lustre: server umount lustre-MDT0001 complete [ 1786.526436] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 1790.289221] Key type lgssc unregistered [ 1790.698817] LNet: 55425:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1790.710898] LNetError: 55425:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1790.731911] LNet: Removed LNI 192.168.204.120@tcp [ 1791.814821] Key type .llcrypt unregistered [ 1791.817317] Key type ._llcrypt unregistered [ 1837.407754] Key type ._llcrypt registered [ 1837.409306] Key type .llcrypt registered [ 1837.505524] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 1838.827375] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1838.853186] alg: No test for adler32 (adler32-zlib) [ 1839.967146] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 1840.172079] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 1841.832233] Key type lgssc registered [ 1842.799677] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1851.251483] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1851.270501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1852.536368] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1852.563549] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1852.711816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1858.538315] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1868.708810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1868.795871] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1868.834144] Lustre: 57634:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1869.113556] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1874.413908] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1883.661736] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1890.766988] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1899.122547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1899.394846] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1899.719306] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1906.937792] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 1908.893893] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1916.100440] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1923.701947] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-27 8h07m27s [ 1925.981438] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1934.696906] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1938.067743] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1938.333707] LustreError: 57643:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1958.805241] Lustre: lustre-MDT0000: Client 4cbd8246-18c2-4bbd-9fc3-c6d2d02d3c0d (at 192.168.204.20@tcp) reconnecting [ 1978.423215] LustreError: 57643:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1981.167566] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1984.422587] Lustre: DEBUG MARKER: done [ 1995.748586] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1995.766402] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1995.791304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1996.259679] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1996.266341] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1998.673120] Lustre: server umount lustre-OST0000 complete [ 2006.498646] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2006.539259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2006.552619] Lustre: Skipped 3 previous similar messages [ 2007.652236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2007.660742] Lustre: Skipped 1 previous similar message [ 2011.479654] Lustre: server umount lustre-MDT0000 complete [ 2014.598078] LustreError: 60396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832540 with bad export cookie 6972117869482423257 [ 2014.602113] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2014.614095] LustreError: 60396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2014.906626] Lustre: server umount lustre-MDT0001 complete [ 2031.539351] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 2035.957308] Key type lgssc unregistered [ 2036.175446] LNet: 61996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2036.186358] LNetError: 61996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2037.221362] LNet: Removed LNI 192.168.204.120@tcp [ 2038.085586] Key type .llcrypt unregistered [ 2038.088542] Key type ._llcrypt unregistered [ 2096.635958] Key type ._llcrypt registered [ 2096.641446] Key type .llcrypt registered [ 2096.879931] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 2098.695078] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2098.741592] alg: No test for adler32 (adler32-zlib) [ 2099.916631] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 2100.132211] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 2101.847979] Key type lgssc registered [ 2103.467814] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2108.803232] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2108.816439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2110.173175] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2110.208231] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2110.377266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2111.385240] Lustre: Failing over lustre-MDT0000 [ 2111.780961] Lustre: server umount lustre-MDT0000 complete [ 2121.570846] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 08:10:45 (1787832645) [ 2130.972441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2131.439616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2136.708926] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2146.504166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2146.621771] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2146.667140] Lustre: 64797:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2147.126276] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2152.125254] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2161.275663] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2168.206544] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2176.925903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2177.123307] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2177.372712] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2184.548298] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2185.922949] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2194.186844] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2201.323940] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2211.809322] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2211.810146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2211.829667] Lustre: Skipped 2 previous similar messages [ 2211.850644] Lustre: Skipped 2 previous similar messages [ 2216.431499] Lustre: server umount lustre-MDT0000 complete [ 2220.043744] LustreError: 64792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832746 with bad export cookie 17316257956636524473 [ 2220.048128] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2220.054949] LustreError: 64792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2220.267362] Lustre: server umount lustre-MDT0001 complete [ 2223.738519] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2228.570760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2238.120612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2238.431198] Lustre: 63720:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787832748/real 1787832748] req@000000002799559c x1874678383683456/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787832764 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2238.457994] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2241.567173] Lustre: 63720:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787832751/real 1787832751] req@00000000e74b6a6f x1874678383683584/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787832767 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2243.552301] Lustre: 63722:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787832753/real 1787832753] req@00000000334c5d62 x1874678383683712/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787832769 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2245.665233] LustreError: 63718:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000334c5d62 x1874678383684480/t0(0) o250->MGC192.168.204.120@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2245.850371] LustreError: 66647:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2245.865278] LustreError: 66647:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2245.902060] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2245.978114] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2246.880801] Lustre: 63722:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787832756/real 1787832756] req@0000000073e1ffb8 x1874678383683968/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787832772 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2247.908074] LustreError: 66648:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2248.932235] LustreError: 66647:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2248.949187] LustreError: 66647:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2249.685347] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2251.999231] Lustre: 63719:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787832761/real 1787832761] req@000000000d1e43fa x1874678383684096/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787832777 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2254.312144] LustreError: 66648:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2254.336679] LustreError: 66648:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2258.558394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2258.914310] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2263.799463] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2264.048650] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2267.114708] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2273.362377] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2279.961114] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2289.642451] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2289.647392] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2289.675595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2292.706561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2292.712371] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2292.725424] Lustre: Skipped 1 previous similar message [ 2295.681151] Lustre: server umount lustre-MDT0000 complete [ 2297.825160] LustreError: 66647:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2297.863474] LustreError: 66647:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2300.868340] LustreError: 64791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832826 with bad export cookie 17316257956636526748 [ 2300.870832] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2300.886141] LustreError: 64791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2301.705828] Lustre: server umount lustre-MDT0001 complete [ 2309.570478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2318.320721] Lustre: 63722:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787832828/real 1787832828] req@0000000048de39bf x1874678383713408/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787832844 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2318.356214] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2320.619883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2326.094912] LustreError: 69306:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2326.169851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2326.248992] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2331.304062] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2340.583726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2344.907228] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2346.490258] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2353.381647] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2361.032354] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2372.067230] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2372.082682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2372.102514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2372.114798] Lustre: Skipped 2 previous similar messages [ 2375.667994] Lustre: server umount lustre-MDT0000 complete [ 2378.213216] LustreError: 71258:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2378.246740] LustreError: 71258:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 2379.945636] LustreError: 64790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832905 with bad export cookie 17316257956636528127 [ 2379.952475] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2379.959117] LustreError: 64790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2380.509473] Lustre: server umount lustre-MDT0001 complete [ 2389.746501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2394.933606] Lustre: DEBUG MARKER: files compared the same [ 2408.067457] Lustre: server umount lustre-OST0000 complete [ 2428.589657] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 2431.936507] Key type lgssc unregistered [ 2432.276653] LNet: 74537:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2432.289927] LNetError: 74537:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2432.310450] LNet: Removed LNI 192.168.204.120@tcp [ 2433.346565] Key type .llcrypt unregistered [ 2433.350481] Key type ._llcrypt unregistered [ 2460.818856] Key type ._llcrypt registered [ 2460.821924] Key type .llcrypt registered [ 2460.938511] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 2462.632494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2462.670629] alg: No test for adler32 (adler32-zlib) [ 2463.927394] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 2464.309230] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 2466.055793] Key type lgssc registered [ 2467.578934] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2476.633992] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2476.652775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2478.329307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2483.878459] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug +malloc all [ 2488.799902] LustreError: 75956:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2488.825921] LustreError: 75956:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2493.932204] LustreError: 75957:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2495.211848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2495.668057] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2500.847964] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug +malloc all [ 2510.562052] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2518.027628] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2526.898073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2527.589322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2535.634051] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug +malloc all [ 2545.889991] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2548.078723] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2563.555732] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2563.563237] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2563.580212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2563.614296] Lustre: Skipped 1 previous similar message [ 2568.004186] Lustre: server umount lustre-OST0000 complete [ 2572.768121] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2572.777453] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2572.807170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2572.823263] Lustre: Skipped 1 previous similar message [ 2573.794513] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2573.816948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2573.829991] Lustre: Skipped 1 previous similar message [ 2578.546977] Lustre: server umount lustre-MDT0000 complete [ 2578.916928] LustreError: 75957:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2578.942191] LustreError: 75957:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2582.773391] LustreError: 75937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833108 with bad export cookie 14253424956409040748 [ 2582.782633] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2583.174383] Lustre: server umount lustre-MDT0001 complete [ 2595.535466] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 2598.727595] Key type lgssc unregistered [ 2599.095704] LNet: 79512:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2599.106588] LNetError: 79512:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2599.129695] LNet: Removed LNI 192.168.204.120@tcp [ 2600.189098] Key type .llcrypt unregistered [ 2600.191564] Key type ._llcrypt unregistered [ 2625.028736] Key type ._llcrypt registered [ 2625.031205] Key type .llcrypt registered [ 2625.142481] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 2626.306192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2626.503987] alg: No test for adler32 (adler32-zlib) [ 2627.580540] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 2627.811449] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 2629.575172] Key type lgssc registered [ 2630.867688] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2639.768183] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2639.806782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2655.458563] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2660.575602] LustreError: 80905:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.120@tcp: failed processing log, type 4: rc = -110 [ 2687.327192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2693.279601] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2702.385769] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2711.224470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2711.334798] Lustre: *** cfs_fail_loc=706, val=0*** [ 2711.864148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2711.965930] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2716.447879] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2717.167564] LustreError: 80931:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2717.179426] LustreError: 80931:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2722.287861] LustreError: 80930:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2726.097927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2726.707205] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2731.819232] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2740.095565] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2747.217703] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2752.488477] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2752.495579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2752.500526] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2752.511058] Lustre: Skipped 1 previous similar message [ 2755.886221] Lustre: server umount lustre-OST0000 complete [ 2762.730973] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2762.752985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2762.760716] Lustre: Skipped 2 previous similar messages [ 2766.622697] Lustre: server umount lustre-MDT0000 complete [ 2767.841406] LustreError: 81995:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2767.855279] LustreError: 81995:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2770.505721] LustreError: 80913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833296 with bad export cookie 15896999443707941255 [ 2770.509397] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2770.519373] LustreError: 80913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2770.909444] Lustre: server umount lustre-MDT0001 complete [ 2781.985160] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 2785.198895] Key type lgssc unregistered [ 2785.546261] LNet: 84525:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2785.553906] LNetError: 84525:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2785.571923] LNet: Removed LNI 192.168.204.120@tcp [ 2786.625928] Key type .llcrypt unregistered [ 2786.628495] Key type ._llcrypt unregistered [ 2816.848333] Key type ._llcrypt registered [ 2816.850672] Key type .llcrypt registered [ 2817.020597] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 2818.229277] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2818.337391] alg: No test for adler32 (adler32-zlib) [ 2819.470592] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 2819.764575] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 2821.447195] Key type lgssc registered [ 2823.256207] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2833.448678] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2833.505921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2839.436214] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2847.755264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2848.389661] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2852.631515] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2853.862315] LustreError: 86630:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2853.894407] LustreError: 86630:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2858.979511] LustreError: 86633:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2858.999507] LustreError: 86633:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2862.893931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2863.440167] LustreError: 86634:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2863.475769] LustreError: 86634:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2863.483267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2868.705461] LustreError: 86633:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2868.721366] LustreError: 86633:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2870.705960] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2873.827474] LustreError: 86629:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2873.840144] LustreError: 86629:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2880.244506] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2880.267024] Lustre: MGS: Client 2ed2127e-a1e6-4fd2-ae1c-0c6de809aba7 (at 0@lo) reconnecting [ 2880.301226] LustreError: 86644:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2880.622672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2880.792364] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2886.087363] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 2906.087267] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2906.101295] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2906.145578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2906.596187] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2909.853660] Lustre: server umount lustre-OST0000 complete [ 2916.833860] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2916.839854] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2916.854295] Lustre: Skipped 1 previous similar message [ 2916.860514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2916.876240] Lustre: Skipped 2 previous similar messages [ 2919.644496] Lustre: server umount lustre-MDT0000 complete [ 2921.979812] LustreError: 87850:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2921.992857] LustreError: 87850:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2924.090262] Lustre: server umount lustre-MDT0001 complete [ 2927.688472] Lustre: server umount MGS complete [ 2940.028712] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 08:24:24 (1787833464) [ 2971.799157] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 2981.981543] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 3018.034917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3024.798607] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3036.537606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3036.934171] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3037.049502] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3037.183243] Lustre: lustre-MDT0001: new disk, initializing [ 3037.385421] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3037.450726] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3037.463306] Lustre: 93827:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3037.480265] LustreError: 93827:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3042.939996] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3055.121508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3055.345162] Lustre: lustre-OST0000: new disk, initializing [ 3055.348220] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3055.355413] Lustre: 94593:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3055.442800] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3062.368581] Lustre: 94885:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000d2c794e8 x1874679137609728/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787833604 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3062.538370] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3073.434371] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3081.507190] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3081.524640] Lustre: MGS: Client 3e824692-f43f-4c41-9612-2e74476d947e (at 0@lo) reconnecting [ 3081.574211] Lustre: 93096:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3081.585745] LustreError: 93813:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3081.618060] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3081.739069] Lustre: lustre-MDT0000: new disk, initializing [ 3081.959281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3082.012763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3082.060981] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3083.111264] Lustre: 95662:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000794bf16f x1874679137616768/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787833625 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3086.982721] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3088.289066] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3092.204431] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3092.219234] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3092.436169] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 3118.560524] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3118.576429] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3118.594498] Lustre: Skipped 1 previous similar message [ 3118.619053] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3122.665747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3122.670321] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3124.437827] Lustre: server umount lustre-OST0000 complete [ 3132.896733] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3132.897876] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3132.905980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3132.911114] Lustre: Skipped 1 previous similar message [ 3132.917979] Lustre: Skipped 2 previous similar messages [ 3136.397335] Lustre: server umount lustre-MDT0000 complete [ 3138.022054] LustreError: 93799:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3138.035208] LustreError: 93799:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3147.557614] Lustre: server umount lustre-MDT0001 complete [ 3152.234364] Lustre: server umount MGS complete [ 3165.779277] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 08:28:09 (1787833689) [ 3186.711946] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 3189.761150] Key type lgssc unregistered [ 3190.061983] LNet: 98164:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3190.076951] LNetError: 98164:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3190.099128] LNet: Removed LNI 192.168.204.120@tcp [ 3191.094493] Key type .llcrypt unregistered [ 3191.096030] Key type ._llcrypt unregistered [ 3213.834218] Key type ._llcrypt registered [ 3213.836024] Key type .llcrypt registered [ 3213.983954] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 3215.304201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3215.481990] alg: No test for adler32 (adler32-zlib) [ 3216.664641] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 3216.821719] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 3218.479349] Key type lgssc registered [ 3220.240571] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3229.354365] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3229.387513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3231.111942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3236.463623] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3241.442706] LustreError: 99388:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3241.465476] LustreError: 99388:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3246.577311] LustreError: 99389:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3246.762395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3247.096446] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3252.032936] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3260.442794] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3266.649521] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3273.999803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3274.338636] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3276.409435] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3280.815674] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3288.432417] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3302.370495] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3302.372936] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3302.393330] LustreError: Skipped 1 previous similar message [ 3302.434465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3302.461705] Lustre: Skipped 1 previous similar message [ 3306.518498] Lustre: server umount lustre-OST0000 complete [ 3312.614343] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3312.616412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3312.628542] Lustre: Skipped 2 previous similar messages [ 3317.472656] Lustre: server umount lustre-MDT0000 complete [ 3317.748693] LustreError: 99388:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3317.785239] LustreError: 99388:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3321.500231] LustreError: 99368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833847 with bad export cookie 11022802131166307913 [ 3321.508951] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3321.514438] LustreError: 99368:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3321.903727] Lustre: server umount lustre-MDT0001 complete [ 3332.361210] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 3335.486666] Key type lgssc unregistered [ 3335.806835] LNet: 102928:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3335.812213] LNetError: 102928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3335.831421] LNet: Removed LNI 192.168.204.120@tcp [ 3336.525514] Key type .llcrypt unregistered [ 3336.527664] Key type ._llcrypt unregistered [ 3354.232422] Key type ._llcrypt registered [ 3354.234642] Key type .llcrypt registered [ 3354.388112] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 3355.890933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3355.990700] alg: No test for adler32 (adler32-zlib) [ 3357.074080] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 3357.400075] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 3359.159168] Key type lgssc registered [ 3360.344696] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3367.233529] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3367.238661] LustreError: 104011:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3367.247657] LustreError: 104013:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3367.253677] LustreError: 104011:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4991 [ 3367.262085] LustreError: 104011:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3367.279287] LustreError: 104011:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3367.295174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3368.908093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3378.064458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3378.148213] LustreError: 104051:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3378.178203] LustreError: 104051:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3378.367990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3384.316104] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3391.427624] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing lsmod [ 3395.057269] LustreError: 105216:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3395.100683] LustreError: 105220:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3395.115212] LustreError: 105216:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4946 [ 3395.120065] LustreError: 105216:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3395.136787] LustreError: 105216:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3395.159979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3395.737754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3399.137194] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3399.151235] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3399.173396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3401.184996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3401.191822] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3401.191822] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3401.197078] Lustre: Skipped 1 previous similar message [ 3403.297692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3405.288199] Lustre: server umount lustre-MDT0000 complete [ 3408.361505] LustreError: 104046:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3408.394133] LustreError: 104046:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3409.210486] LustreError: 104029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833935 with bad export cookie 1502517290283641760 [ 3409.230639] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3409.479317] LustreError: 104481:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000000ab1b739 x1874679701721728/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3409.649518] Lustre: server umount lustre-MDT0001 complete [ 3423.892263] Lustre: server umount lustre-OST0000 complete [ 3432.876775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3433.479851] LustreError: 106283:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3433.558586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3438.563906] LustreError: 106284:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3438.629665] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3443.684939] LustreError: 106283:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3448.693970] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3449.067782] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3453.845488] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3460.659660] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3465.656787] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3472.447751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3472.886377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3478.712828] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3479.038589] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3486.804473] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3499.491824] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3499.497647] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3499.504533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3499.504546] Lustre: Skipped 2 previous similar messages [ 3499.507695] LustreError: Skipped 1 previous similar message [ 3499.556279] Lustre: Skipped 1 previous similar message [ 3501.617309] Lustre: server umount lustre-OST0000 complete [ 3509.734158] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3509.753148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3509.760128] Lustre: Skipped 2 previous similar messages [ 3512.279792] Lustre: server umount lustre-MDT0000 complete [ 3514.849202] LustreError: 106284:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3514.875685] LustreError: 106284:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3516.869434] LustreError: 108127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834042 with bad export cookie 1502517290283642460 [ 3516.888447] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3516.891504] LustreError: 108127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3517.401826] Lustre: server umount lustre-MDT0001 complete [ 3528.852535] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 3532.043544] Key type lgssc unregistered [ 3532.448153] LNet: 109819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3532.456178] LNetError: 109819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3532.489929] LNet: Removed LNI 192.168.204.120@tcp [ 3533.618668] Key type .llcrypt unregistered [ 3533.620661] Key type ._llcrypt unregistered [ 3560.618680] Key type ._llcrypt registered [ 3560.621384] Key type .llcrypt registered [ 3560.739310] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 3562.005336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3562.122865] alg: No test for adler32 (adler32-zlib) [ 3563.259311] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 3563.580163] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 3565.392170] Key type lgssc registered [ 3566.580821] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3575.727150] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3575.742863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3577.115937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3581.853597] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3587.558710] LustreError: 111237:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3587.599462] LustreError: 111237:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3592.684148] LustreError: 111236:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3593.342948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3593.960282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3598.943905] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3609.798372] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3617.047743] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3626.835062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3627.299979] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3635.979561] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3645.349758] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3647.808238] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3657.428883] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3662.297735] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3669.473581] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3669.486235] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3669.505123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3673.579091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3673.585864] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3673.597046] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3674.939248] Lustre: server umount lustre-OST0000 complete [ 3681.249805] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3681.265232] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3681.281538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3681.286797] Lustre: Skipped 1 previous similar message [ 3683.810708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3683.811856] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3687.695158] Lustre: server umount lustre-MDT0000 complete [ 3688.929060] LustreError: 112484:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3688.968831] LustreError: 112484:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3692.538100] LustreError: 111216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834218 with bad export cookie 1959345061012429960 [ 3692.548827] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3692.559055] LustreError: 111216:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3693.055194] Lustre: server umount lustre-MDT0001 complete [ 3705.627982] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 3708.479571] Key type lgssc unregistered [ 3708.771914] LNet: 114907:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3708.787356] LNetError: 114907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3708.803880] LNet: Removed LNI 192.168.204.120@tcp [ 3709.661160] Key type .llcrypt unregistered [ 3709.665936] Key type ._llcrypt unregistered [ 3727.346037] Key type ._llcrypt registered [ 3727.348172] Key type .llcrypt registered [ 3727.493122] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 3728.466719] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3728.491728] alg: No test for adler32 (adler32-zlib) [ 3729.673418] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 3729.988305] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 3731.657158] Key type lgssc registered [ 3733.078363] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3743.659381] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3743.706631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3745.411414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3752.441498] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3756.010167] LustreError: 116131:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3756.049586] LustreError: 116131:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3761.122349] LustreError: 116130:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3765.318822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3765.729822] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3772.241242] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3781.101474] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3788.668123] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3798.461475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3799.084269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3807.645880] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3815.430523] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3815.992835] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3825.651635] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3825.653365] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3825.655121] LustreError: Skipped 1 previous similar message [ 3825.662940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3825.680539] Lustre: Skipped 1 previous similar message [ 3825.699552] Lustre: Skipped 1 previous similar message [ 3830.774772] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3830.796496] Lustre: Skipped 1 previous similar message [ 3831.026590] Lustre: server umount lustre-OST0000 complete [ 3837.410829] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3837.426253] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3837.444684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3840.993073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3840.994183] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3842.602741] Lustre: server umount lustre-MDT0000 complete [ 3846.126262] LustreError: 116130:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3846.146682] LustreError: 116130:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3847.392153] LustreError: 117973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834373 with bad export cookie 9314104599123050929 [ 3847.411551] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3847.415717] LustreError: 117973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3848.212438] Lustre: server umount lustre-MDT0001 complete [ 3864.251876] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 3868.680777] Key type lgssc unregistered [ 3869.118542] LNet: 119670:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3869.129736] LNetError: 119670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3869.163434] LNet: Removed LNI 192.168.204.120@tcp [ 3870.260214] Key type .llcrypt unregistered [ 3870.262677] Key type ._llcrypt unregistered [ 3901.069627] Key type ._llcrypt registered [ 3901.073059] Key type .llcrypt registered [ 3901.188990] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 3901.922027] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3901.991521] alg: No test for adler32 (adler32-zlib) [ 3903.041280] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 3903.224864] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 3904.911525] Key type lgssc registered [ 3906.828983] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3917.962240] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3918.002619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3919.822623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3929.071150] LustreError: 121094:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3929.110471] LustreError: 121094:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3929.299382] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3934.189264] LustreError: 121093:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3939.302093] LustreError: 121094:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3941.995514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3942.700311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3948.812590] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3958.109468] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3965.516679] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3974.481088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3974.925223] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3983.299787] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 3989.299577] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3991.828977] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4000.017526] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 4004.540093] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 4004.545516] Lustre: Skipped 2 previous similar messages [ 4009.583221] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 4009.616550] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 4009.630888] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 4009.635745] Lustre: Skipped 2 previous similar messages [ 4015.178583] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4015.181609] Lustre: Skipped 1 previous similar message [ 4021.681814] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.20@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 4021.689418] Lustre: Skipped 2 previous similar messages [ 4023.619430] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.20@tcp 192.168.0.[2,10]@tcp [ 4029.531307] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.20/24@tcp in log lustre-MDT0000 [ 4029.536726] Lustre: Skipped 2 previous similar messages [ 4038.218548] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.20/24@tcp [ 4038.227211] Lustre: Skipped 1 previous similar message [ 4045.792679] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4045.802078] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4045.818901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4045.823037] Lustre: Skipped 1 previous similar message [ 4045.843808] Lustre: Skipped 1 previous similar message [ 4050.391106] Lustre: server umount lustre-OST0000 complete [ 4055.521139] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4055.544805] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4055.573579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4061.157365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4061.165663] Lustre: Skipped 3 previous similar messages [ 4065.250320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4065.257691] Lustre: Skipped 1 previous similar message [ 4069.343744] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4069.615944] Lustre: server umount lustre-MDT0000 complete [ 4070.371907] LustreError: 122125:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4070.402714] LustreError: 122125:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4073.717918] LustreError: 121074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834599 with bad export cookie 3256119929346534655 [ 4073.724109] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4073.732644] LustreError: 121074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4074.158416] Lustre: server umount lustre-MDT0001 complete [ 4087.290201] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 4091.037139] Key type lgssc unregistered [ 4091.460074] LNet: 126046:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4091.475089] LNetError: 126046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4091.506303] LNet: Removed LNI 192.168.204.120@tcp [ 4092.607903] Key type .llcrypt unregistered [ 4092.610281] Key type ._llcrypt unregistered [ 4125.754576] Key type ._llcrypt registered [ 4125.756711] Key type .llcrypt registered [ 4125.887326] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 4126.805449] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4127.068689] alg: No test for adler32 (adler32-zlib) [ 4128.196748] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 4128.629141] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 4130.423667] Key type lgssc registered [ 4131.916395] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4141.301976] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4141.333618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4142.951926] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4142.995501] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.20/24@tcp [ 4143.053414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4148.474562] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4152.294889] LustreError: 127611:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4152.312531] LustreError: 127611:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4157.422482] LustreError: 127610:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4159.458766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4159.979086] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4160.020817] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.20/24@tcp [ 4160.062337] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4165.942673] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4174.971869] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4180.330487] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4187.365861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4187.622286] LustreError: 129437:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4187.752312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4192.789390] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 4195.009367] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4203.307281] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4215.269434] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4215.274991] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4215.277211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4215.296106] Lustre: Skipped 1 previous similar message [ 4220.400076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4220.406375] Lustre: Skipped 2 previous similar messages [ 4220.800803] Lustre: server umount lustre-OST0000 complete [ 4225.517801] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4225.519959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4225.529924] Lustre: Skipped 1 previous similar message [ 4230.626919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4230.631410] Lustre: Skipped 2 previous similar messages [ 4235.744335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4235.753454] Lustre: Skipped 1 previous similar message [ 4239.327210] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4239.529091] Lustre: server umount lustre-MDT0000 complete [ 4240.868606] LustreError: 127611:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4240.886323] LustreError: 127611:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4243.621913] LustreError: 127590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834769 with bad export cookie 6606516645235971828 [ 4243.633234] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4243.645729] LustreError: 127590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4244.233931] Lustre: server umount lustre-MDT0001 complete [ 4256.689955] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 4260.152155] Key type lgssc unregistered [ 4260.440750] LNet: 131206:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4260.463770] LNetError: 131206:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4260.487241] LNet: Removed LNI 192.168.204.120@tcp [ 4261.496500] Key type .llcrypt unregistered [ 4261.505568] Key type ._llcrypt unregistered [ 4292.291385] Key type ._llcrypt registered [ 4292.300618] Key type .llcrypt registered [ 4292.438069] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 4309.006456] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 4310.326465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4310.489953] alg: No test for adler32 (adler32-zlib) [ 4311.706641] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 4312.133644] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 4313.872287] Key type lgssc registered [ 4315.617769] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4358.260919] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4358.295812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4359.579735] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4359.627795] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4359.759656] Lustre: lustre-MDT0000: new disk, initializing [ 4359.876335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4359.899793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4365.425508] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4377.612023] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4377.721623] Lustre: 134611:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4377.827470] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4377.831440] Lustre: Skipped 1 previous similar message [ 4377.953493] Lustre: lustre-MDT0001: new disk, initializing [ 4378.112343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4378.163494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4378.193446] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4382.216865] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4392.718763] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4400.070121] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4408.576881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4408.783385] Lustre: lustre-OST0000: new disk, initializing [ 4408.787768] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4408.796050] Lustre: 136581:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4408.870948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4414.542359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4414.552196] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4414.668985] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4414.977733] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4424.429364] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4429.601866] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4429.777327] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4434.751680] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4435.053444] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4440.551336] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4440.553284] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4440.573974] LustreError: Skipped 1 previous similar message [ 4440.585820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4440.591615] Lustre: Skipped 1 previous similar message [ 4440.630232] Lustre: Skipped 1 previous similar message [ 4443.459668] Lustre: server umount lustre-OST0000 complete [ 4449.763057] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4449.778376] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4449.799939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4450.795375] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4450.820766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4450.826463] Lustre: Skipped 1 previous similar message [ 4453.626866] Lustre: server umount lustre-MDT0000 complete [ 4457.475280] LustreError: 134602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834983 with bad export cookie 18061400447145894822 [ 4457.483726] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4457.492689] LustreError: 134602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4457.788205] Lustre: server umount lustre-MDT0001 complete [ 4466.088643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4466.598832] LustreError: 138797:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4466.616708] LustreError: 138797:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4466.665636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4471.777339] LustreError: 138798:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4472.491704] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4476.899637] LustreError: 138797:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4482.017203] LustreError: 138798:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4483.104895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4483.457325] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4489.053751] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4497.418884] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4503.937429] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4511.120832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4511.458938] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4518.501546] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4527.364700] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4532.983236] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4537.596496] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 4543.314978] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4543.618253] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4554.648730] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4554.949139] Lustre: lustre-OST0001: new disk, initializing [ 4554.956226] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4554.968818] Lustre: 142199:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4555.062093] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4561.520259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4561.528942] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4561.647592] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4562.039988] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4570.914303] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4571.117186] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4575.599093] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4575.807772] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4582.221872] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d8aa00ea800.ost_server_uuid 50 [ 4584.125987] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8d8aa00ea800.ost_server_uuid in IDLE state after 0 sec [ 4597.729742] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4597.738889] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4597.747690] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4597.764928] Lustre: Skipped 1 previous similar message [ 4597.783272] Lustre: Skipped 1 previous similar message [ 4601.008245] Lustre: server umount lustre-OST0001 complete [ 4602.857570] LustreError: 141872:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4602.880146] LustreError: 141872:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4605.924627] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4605.933260] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4605.951181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4611.320562] Lustre: server umount lustre-OST0000 complete [ 4616.672584] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4616.683460] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4616.697360] Lustre: Skipped 1 previous similar message [ 4616.702247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4616.707216] Lustre: Skipped 2 previous similar messages [ 4621.646402] Lustre: server umount lustre-MDT0000 complete [ 4623.329670] LustreError: 138798:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4623.346608] LustreError: 138798:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4626.018187] LustreError: 142198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835151 with bad export cookie 18061400447145895704 [ 4626.039585] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4626.045578] LustreError: 142198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4626.467396] Lustre: server umount lustre-MDT0001 complete [ 4648.497398] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 4650.905890] Key type lgssc unregistered [ 4651.192559] LNet: 145207:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4651.199237] LNetError: 145207:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4651.218492] LNet: Removed LNI 192.168.204.120@tcp [ 4651.963529] Key type .llcrypt unregistered [ 4651.965601] Key type ._llcrypt unregistered [ 4702.879434] Key type ._llcrypt registered [ 4702.886836] Key type .llcrypt registered [ 4703.040858] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 4719.260560] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 4721.163451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4721.195662] alg: No test for adler32 (adler32-zlib) [ 4722.197429] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 4722.400870] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 4724.055318] Key type lgssc registered [ 4725.407188] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4763.499908] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4763.526804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4764.836464] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4764.903857] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4764.973956] Lustre: lustre-MDT0000: new disk, initializing [ 4765.036349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4765.060855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4770.331431] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4783.083922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4783.204645] Lustre: 149365:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4783.278562] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4783.285112] Lustre: Skipped 1 previous similar message [ 4783.372759] Lustre: lustre-MDT0001: new disk, initializing [ 4783.489487] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4783.539549] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4783.549302] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4788.565833] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4799.361226] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4806.118988] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4813.514676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4813.804457] Lustre: lustre-OST0000: new disk, initializing [ 4813.808967] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4813.827905] Lustre: 151336:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4813.968669] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4821.255472] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4822.662861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4822.681416] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4822.876464] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4832.201771] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4837.652276] Lustre: Failing over lustre-OST0000 [ 4837.863549] Lustre: server umount lustre-OST0000 complete [ 4854.162647] Lustre: 147592:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787835364/real 1787835364] req@00000000f79f17e2 x1874681133040896/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787835380 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4854.228409] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4855.263477] Lustre: 147592:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787835365/real 1787835365] req@00000000be619880 x1874681133041920/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1787835381 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4855.272749] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4855.296166] Lustre: 147592:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4859.420390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4859.423402] Lustre: 147594:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787835369/real 1787835369] req@0000000005f27426 x1874681133042816/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787835385 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4860.006494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4860.041096] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4864.097438] Lustre: 147591:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787835374/real 1787835374] req@0000000089c1fa11 x1874681133044096/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787835390 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4864.124883] Lustre: 147591:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4864.134652] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4867.001741] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 4867.003139] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4867.742467] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4869.090277] Lustre: 147591:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787835379/real 1787835379] req@000000003a72b3fd x1874681133045120/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787835395 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4869.121428] Lustre: 147591:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4871.814040] Lustre: Failing over lustre-MDT0000 [ 4872.226697] Lustre: server umount lustre-MDT0000 complete [ 4874.722963] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4874.746631] Lustre: Skipped 2 previous similar messages [ 4877.155727] LustreError: 151605:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4877.184172] LustreError: 151605:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4879.843450] LustreError: 149378:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4879.856334] LustreError: 149378:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4882.275704] LustreError: 149375:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4884.968178] LustreError: 151605:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4885.001514] LustreError: 151605:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4890.081423] Lustre: 147592:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787835400/real 1787835400] req@00000000324e7a63 x1874681133058048/t0(0) o400->MGC192.168.204.120@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787835416 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4890.083672] LustreError: 149379:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4890.099269] Lustre: 147592:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4890.111934] LustreError: 149379:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4890.130273] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4892.377324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4898.658423] LustreError: 149375:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4898.683706] LustreError: 149375:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4900.322619] LustreError: 147590:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d9342485 x1874681133062912/t0(0) o250->MGC192.168.204.120@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4900.691800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4900.759190] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4903.777177] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4905.242884] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 4905.964385] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4905.967600] Lustre: Skipped 1 previous similar message [ 4905.984879] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 4906.044550] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4916.199615] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4916.201991] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4916.221818] Lustre: Skipped 1 previous similar message [ 4919.488753] Lustre: server umount lustre-OST0000 complete [ 4926.433270] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4926.433804] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4926.440020] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4926.440040] Lustre: Skipped 1 previous similar message [ 4926.445121] Lustre: Skipped 1 previous similar message [ 4929.277621] Lustre: server umount lustre-MDT0000 complete [ 4931.558179] LustreError: 150532:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4931.584695] LustreError: 150532:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4933.324763] LustreError: 149359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835459 with bad export cookie 5266556157401488036 [ 4933.335220] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4933.336292] LustreError: 149359:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4933.749680] Lustre: server umount lustre-MDT0001 complete [ 4945.006978] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 4947.919178] Key type lgssc unregistered [ 4948.241917] LNet: 154993:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4948.254467] LNetError: 154993:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4948.267396] LNet: Removed LNI 192.168.204.120@tcp [ 4949.263727] Key type .llcrypt unregistered [ 4949.272446] Key type ._llcrypt unregistered [ 4984.057624] Key type ._llcrypt registered [ 4984.059822] Key type .llcrypt registered [ 4984.179773] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 4984.889627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4984.904179] alg: No test for adler32 (adler32-zlib) [ 4985.880585] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 4986.119368] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 4987.815305] Key type lgssc registered [ 4989.192429] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4997.619916] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4997.656665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4999.093636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5003.109798] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5009.384064] LustreError: 156514:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5009.395595] LustreError: 156514:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5014.499338] LustreError: 156515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5015.741946] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5019.626169] LustreError: 156515:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5020.508360] LustreError: 156543:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5020.530764] LustreError: 156543:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5020.552889] LustreError: 156543:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 21, retries 0, failed: rc = -5 [ 5020.929520] Lustre: server umount lustre-MDT0000 complete [ 5038.450975] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 5048.126562] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 5089.241910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5089.530073] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5089.568561] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5089.637055] Lustre: lustre-MDT0000: new disk, initializing [ 5089.718657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5089.731916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5095.141355] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5107.044684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5107.158852] Lustre: 159850:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5107.245847] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5107.250992] Lustre: Skipped 1 previous similar message [ 5107.356795] Lustre: lustre-MDT0001: new disk, initializing [ 5107.465754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5107.521592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5107.533720] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5112.493594] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5122.043653] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5128.191724] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5134.975967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5135.359347] Lustre: lustre-OST0000: new disk, initializing [ 5135.369209] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5135.378377] Lustre: 161827:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5135.467823] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5141.710670] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5143.111026] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5143.121454] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5143.254750] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5151.547592] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5155.578570] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5157.386517] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 09:01:21 (1787835681) [ 5440.480266] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5440.488960] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5440.525362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5443.551922] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5443.552602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5443.570533] Lustre: Skipped 1 previous similar message [ 5443.590972] Lustre: Skipped 2 previous similar messages [ 5445.174291] Lustre: server umount lustre-MDT0000 complete [ 5448.672936] LustreError: 159857:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5448.700604] LustreError: 159857:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5449.130758] LustreError: 160645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835975 with bad export cookie 17469617946999060094 [ 5449.130988] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5449.135528] LustreError: 160645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5453.813890] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5453.819199] LustreError: 160923:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5453.844535] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5455.610695] Lustre: server umount lustre-MDT0001 complete [ 5474.271283] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5474.497411] Lustre: server umount lustre-OST0000 complete [ 5486.516086] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 5498.564885] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 5540.244623] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5540.576772] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5540.618537] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5540.729274] Lustre: lustre-MDT0000: new disk, initializing [ 5540.816563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5540.843300] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5546.204871] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5558.717132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5558.860670] Lustre: 166501:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5558.905786] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5558.908941] Lustre: Skipped 1 previous similar message [ 5559.036636] Lustre: lustre-MDT0001: new disk, initializing [ 5559.120978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5559.159631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5559.173379] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5564.633673] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5577.086541] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5584.362680] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5594.444506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5594.719682] Lustre: lustre-OST0000: new disk, initializing [ 5594.725456] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5594.733263] Lustre: 168472:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5594.824356] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5601.047276] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5604.481229] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5604.497044] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5604.629818] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5611.909854] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5616.229805] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5616.415445] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5619.898142] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5620.081311] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5625.321634] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5625.323194] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5625.330981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5625.330991] Lustre: Skipped 1 previous similar message [ 5625.404534] Lustre: Skipped 1 previous similar message [ 5627.823458] Lustre: server umount lustre-OST0000 complete [ 5630.944614] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5630.953353] LustreError: Skipped 1 previous similar message [ 5630.957774] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5630.969843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5630.981624] Lustre: Skipped 1 previous similar message [ 5640.672564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5640.687361] Lustre: Skipped 3 previous similar messages [ 5645.281216] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5645.496621] Lustre: server umount lustre-MDT0000 complete [ 5645.795203] LustreError: 168332:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5645.819383] LustreError: 168332:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5649.669610] LustreError: 166491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836175 with bad export cookie 17469617946999218924 [ 5649.683200] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5649.687016] LustreError: 166491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5650.130281] Lustre: server umount lustre-MDT0001 complete [ 5660.914712] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 09:09:44 (1787836184) [ 5678.825860] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 5690.795376] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 5733.192681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5733.441925] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5733.464329] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5733.534730] Lustre: lustre-MDT0000: new disk, initializing [ 5733.595735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5733.620733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5738.822140] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5750.680179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5750.860890] Lustre: 173226:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5751.154850] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5755.620301] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5767.010180] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5775.436955] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5786.181737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5786.698710] Lustre: lustre-OST0000: new disk, initializing [ 5786.706788] Lustre: Skipped 1 previous similar message [ 5786.718896] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5786.727733] Lustre: Skipped 2 previous similar messages [ 5786.749215] Lustre: 175209:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5792.869349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5792.889130] Lustre: Skipped 1 previous similar message [ 5792.900605] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5793.178491] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5796.439438] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5808.655615] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5818.857460] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5818.858639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5818.867227] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5818.876751] Lustre: Skipped 2 previous similar messages [ 5818.893060] Lustre: Skipped 1 previous similar message [ 5823.039553] Lustre: server umount lustre-OST0000 complete [ 5828.064355] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5841.376275] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5841.494733] Lustre: server umount lustre-MDT0000 complete [ 5843.424571] LustreError: 173239:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5843.444815] LustreError: 173239:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5845.698787] LustreError: 173221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836371 with bad export cookie 17469617946999219806 [ 5845.701430] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5845.704048] LustreError: 173221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5845.902671] Lustre: server umount lustre-MDT0001 complete [ 5856.732569] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 09:13:00 (1787836380) [ 5877.793773] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 5890.019425] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 5930.384991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5930.916528] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5930.948820] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5931.076285] Lustre: lustre-MDT0000: new disk, initializing [ 5931.175625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5931.177844] Lustre: Skipped 2 previous similar messages [ 5931.197381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5937.138044] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5949.422301] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5949.582852] Lustre: 179532:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5949.892787] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5955.249204] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5965.527693] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5972.401755] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5980.565780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5980.792514] Lustre: 181513:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5988.053312] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 5988.454023] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5988.633850] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6000.451993] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6009.319953] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6009.325660] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6009.329391] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6009.332028] Lustre: Skipped 3 previous similar messages [ 6009.344331] Lustre: Skipped 8 previous similar messages [ 6015.160857] Lustre: server umount lustre-OST0000 complete [ 6025.805671] Lustre: server umount lustre-MDT0000 complete [ 6029.800047] LustreError: 180666:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6029.820595] LustreError: 180666:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6030.065630] LustreError: 181512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836556 with bad export cookie 17469617946999220891 [ 6030.068025] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6030.072783] LustreError: 181512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6030.416452] Lustre: server umount lustre-MDT0001 complete [ 6045.420477] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 6048.644789] Key type lgssc unregistered [ 6048.910823] LNet: 183372:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6048.922246] LNetError: 183372:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6048.949293] LNet: Removed LNI 192.168.204.120@tcp [ 6049.817397] Key type .llcrypt unregistered [ 6049.825353] Key type ._llcrypt unregistered [ 6080.680399] Key type ._llcrypt registered [ 6080.682448] Key type .llcrypt registered [ 6080.792478] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 6082.272241] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6082.338159] alg: No test for adler32 (adler32-zlib) [ 6083.468384] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 6083.702267] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 6085.327131] Key type lgssc registered [ 6086.702352] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6097.619927] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6097.666508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6099.368055] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6105.119567] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6109.669381] LustreError: 184794:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6109.703284] LustreError: 184794:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6110.695976] LustreError: 184795:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6115.809219] LustreError: 184794:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6120.931446] LustreError: 184795:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6121.732671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6122.281626] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6128.286823] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6138.512090] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6147.227926] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6156.446824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6157.260853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6164.529168] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6170.638722] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6175.966825] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6196.203491] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6196.214888] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6196.238540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6196.244051] Lustre: Skipped 1 previous similar message [ 6201.313853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6201.320936] Lustre: Skipped 1 previous similar message [ 6201.516878] Lustre: server umount lustre-OST0000 complete [ 6206.433539] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6206.447597] Lustre: Skipped 1 previous similar message [ 6206.451596] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6206.460139] Lustre: Skipped 1 previous similar message [ 6211.554853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6216.676082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6216.681395] Lustre: Skipped 2 previous similar messages [ 6220.255153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6220.508773] Lustre: server umount lustre-MDT0000 complete [ 6221.802717] LustreError: 185523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6221.828749] LustreError: 185523:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6225.072934] LustreError: 184774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836751 with bad export cookie 1225389630185947995 [ 6225.083112] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6225.093545] LustreError: 184774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6225.337695] Lustre: server umount lustre-MDT0001 complete [ 6241.614206] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 6245.044357] Key type lgssc unregistered [ 6245.477115] LNet: 188357:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6245.490412] LNetError: 188357:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6245.520455] LNet: Removed LNI 192.168.204.120@tcp [ 6246.598402] Key type .llcrypt unregistered [ 6246.601447] Key type ._llcrypt unregistered [ 6274.184767] Key type ._llcrypt registered [ 6274.190800] Key type .llcrypt registered [ 6274.311454] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 6275.461533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6275.683044] alg: No test for adler32 (adler32-zlib) [ 6276.790884] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 6277.141545] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 6278.855211] Key type lgssc registered [ 6280.221815] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6288.358849] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6288.383319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6289.834673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6294.403531] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6299.123712] LustreError: 189782:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6299.149675] LustreError: 189782:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6303.356914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6303.661432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6308.250463] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6316.004990] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6322.517953] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6330.136522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6330.755711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6334.838102] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6338.089680] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6346.376920] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6354.314749] Lustre: lustre-OST0000: Not available for connect from 192.168.204.20@tcp (stopping) [ 6355.425214] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6355.435693] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6355.448286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6358.501493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6358.516500] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6358.517761] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6359.918627] Lustre: server umount lustre-OST0000 complete [ 6367.904452] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8d8a87962000.ost_server_uuid 50 [ 6369.761186] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6385.646400] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6385.657710] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6385.681417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6385.692483] Lustre: Skipped 2 previous similar messages [ 6388.197496] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6390.364677] Lustre: server umount lustre-MDT0000 complete [ 6393.314674] LustreError: 189778:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6393.341559] LustreError: 189778:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6394.648878] LustreError: 189763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836920 with bad export cookie 6091528048165726472 [ 6394.658918] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6394.671902] LustreError: 189763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6395.082324] Lustre: server umount lustre-MDT0001 complete [ 6405.956520] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 09:22:09 (1787836929) [ 6414.178131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6414.658859] LustreError: 193698:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6414.755047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6419.940356] LustreError: 193699:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6420.226559] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6424.040924] LustreError: 193698:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6430.479118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6431.060679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6436.402746] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6445.486945] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6451.862844] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6459.501819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6459.896059] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6465.015408] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6467.114169] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6475.326573] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6482.716729] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6482.902137] Lustre: lustre-OST0001: new disk, initializing [ 6482.905941] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6482.911317] Lustre: 196543:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6483.002365] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6488.196406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6488.209116] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6488.283842] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6489.925900] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6503.046700] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6509.034574] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6509.047241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6509.054166] Lustre: Skipped 3 previous similar messages [ 6519.277271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6519.284542] Lustre: Skipped 5 previous similar messages [ 6521.311169] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6521.366066] Lustre: server umount lustre-OST0000 complete [ 6524.400607] LustreError: 196822:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6524.409650] LustreError: 196822:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6528.755109] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6529.056867] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6533.711220] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6533.940922] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6544.869980] LustreError: 195523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6544.889257] LustreError: 195523:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 6549.984363] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6549.990556] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6550.007776] LustreError: Skipped 1 previous similar message [ 6550.009576] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6550.018643] Lustre: Skipped 2 previous similar messages [ 6550.023611] Lustre: Skipped 1 previous similar message [ 6555.134784] Lustre: server umount lustre-OST0001 complete [ 6565.999438] Lustre: server umount lustre-MDT0000 complete [ 6570.545289] LustreError: 196542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837096 with bad export cookie 6091528048165727536 [ 6570.558350] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6570.566071] LustreError: 196542:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6571.078099] Lustre: server umount lustre-MDT0001 complete [ 6597.910387] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 09:25:22 (1787837122) [ 6605.137590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6605.472890] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6605.537864] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6605.866571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6611.632540] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6622.591173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6622.670544] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6622.733104] Lustre: 199989:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6628.518589] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6637.910466] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6644.744329] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6652.566969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6652.828384] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6659.620939] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6660.735290] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6668.265830] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6676.549959] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6676.642961] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6676.922255] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6676.929658] Lustre: Skipped 2 previous similar messages [ 6684.742781] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6694.081591] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6697.962522] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6697.974828] Lustre: Skipped 2 previous similar messages [ 6697.979874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6697.985111] Lustre: Skipped 5 previous similar messages [ 6704.112706] Lustre: server umount lustre-OST0000 complete [ 6705.510744] LustreError: 201839:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6705.534509] LustreError: 201839:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6723.554602] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6726.656571] Lustre: server umount lustre-OST0001 complete [ 6733.795990] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6733.821982] Lustre: Skipped 4 previous similar messages [ 6737.811307] Lustre: server umount lustre-MDT0000 complete [ 6742.552627] LustreError: 199979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837268 with bad export cookie 6091528048165728950 [ 6742.554627] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6742.564907] LustreError: 199979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6743.058814] Lustre: server umount lustre-MDT0001 complete [ 6771.043542] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 09:28:15 (1787837295) [ 6789.303652] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 6800.277257] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 6842.367972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6842.582417] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6842.626427] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6842.733823] Lustre: lustre-MDT0000: new disk, initializing [ 6842.810885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6842.830567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6847.463397] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6861.053463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6861.251950] Lustre: 208083:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6861.325566] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6861.333421] Lustre: Skipped 1 previous similar message [ 6861.414462] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6861.424422] Lustre: Skipped 7 previous similar messages [ 6861.446614] Lustre: lustre-MDT0001: new disk, initializing [ 6861.598563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6861.617736] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6867.753085] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6880.152805] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6888.621608] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6897.189129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6897.465238] Lustre: lustre-OST0000: new disk, initializing [ 6897.468847] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6897.473524] Lustre: 210055:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6904.770161] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6906.490112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6906.503220] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6906.626866] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6916.178134] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6921.854878] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6922.166718] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6927.827830] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6928.076559] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6932.448654] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6932.456627] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6932.463077] LustreError: Skipped 2 previous similar messages [ 6932.487547] Lustre: Skipped 1 previous similar message [ 6936.666595] Lustre: server umount lustre-OST0000 complete [ 6947.149665] Lustre: server umount lustre-MDT0000 complete [ 6947.814558] LustreError: 208091:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6947.836548] LustreError: 208091:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 6951.470258] LustreError: 208077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837477 with bad export cookie 6091528048165730511 [ 6951.476654] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6951.483323] LustreError: 208077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6951.956905] Lustre: server umount lustre-MDT0001 complete [ 6961.075294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6966.760634] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6977.441291] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6983.433807] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 6993.251967] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7000.482821] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7008.979757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7016.292957] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7025.077980] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7031.355922] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7032.746062] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 7038.294512] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7038.483076] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7042.532995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7042.537886] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7042.556669] Lustre: Skipped 4 previous similar messages [ 7046.874428] Lustre: server umount lustre-OST0000 complete [ 7054.674518] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7061.062778] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7091.244244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7099.540735] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7107.123951] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7107.139391] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7107.143742] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7107.176574] Lustre: Skipped 1 previous similar message [ 7109.886174] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7112.016546] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7122.404034] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7122.420145] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7122.438137] Lustre: Skipped 5 previous similar messages [ 7124.320345] Lustre: server umount lustre-OST0000 complete [ 7143.391170] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7143.621258] Lustre: server umount lustre-MDT0000 complete [ 7148.003764] LustreError: 213972:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7148.036664] LustreError: 213972:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7148.428177] LustreError: 212255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837674 with bad export cookie 6091528048165731393 [ 7148.447124] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7148.983381] Lustre: server umount lustre-MDT0001 complete [ 7159.319511] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 09:34:43 (1787837683) [ 7168.340872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7169.117083] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7169.125487] Lustre: Skipped 6 previous similar messages [ 7174.780613] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7185.124897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7190.552472] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7199.793377] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7207.303208] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7215.287112] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7222.774977] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7230.383102] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7235.176687] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7238.660840] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 7243.538414] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7243.763585] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7250.461673] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7250.674276] Lustre: lustre-OST0001: new disk, initializing [ 7250.677597] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7250.684541] Lustre: 221646:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7257.465615] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7258.834527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7258.850345] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7258.998728] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7269.058752] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7276.686750] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7276.944574] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7282.227573] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7282.497913] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7284.704495] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7284.721194] LustreError: Skipped 2 previous similar messages [ 7290.920296] Lustre: server umount lustre-OST0001 complete [ 7297.700524] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7297.857505] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7302.533713] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7302.855361] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7333.434709] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7338.988088] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7339.002530] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7339.007159] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7341.645789] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7353.089749] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7355.579865] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7359.463943] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7359.476417] Lustre: Skipped 14 previous similar messages [ 7361.264638] Lustre: server umount lustre-OST0001 complete [ 7379.941729] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7379.965666] Lustre: Skipped 7 previous similar messages [ 7394.271275] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7394.450288] Lustre: server umount lustre-OST0000 complete [ 7404.792482] Lustre: server umount lustre-MDT0000 complete [ 7405.544637] LustreError: 224911:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7405.559860] LustreError: 224911:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 40 previous similar messages [ 7408.621158] LustreError: 218235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837934 with bad export cookie 6091528048165732359 [ 7408.626047] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7408.636915] LustreError: 218235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 7415.090917] Lustre: server umount lustre-MDT0001 complete [ 7446.727240] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 09:39:30 (1787837970) [ 7454.661602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7455.075273] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7455.127661] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7461.388616] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7470.391463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7470.461494] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7470.475694] Lustre: 226894:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7476.118600] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7485.654761] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7492.935932] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7500.907018] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7501.198583] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7510.029468] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7519.695124] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7530.700509] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7530.875431] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7539.820688] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7549.014827] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7551.165939] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7557.790634] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7558.084962] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7563.784798] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7563.993552] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7570.889930] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d8a86eef000.ost_server_uuid 50 [ 7572.770444] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8d8a86eef000.ost_server_uuid in IDLE state after 0 sec [ 7574.204257] Lustre: Permanently deactivating lustre-OST0001 [ 7574.226837] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7578.762737] Lustre: Permanently reactivating lustre-OST0001 [ 7578.803975] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7578.813387] Lustre: Skipped 2 previous similar messages [ 7584.762448] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 7584.774777] LustreError: Skipped 3 previous similar messages [ 7590.395857] Lustre: server umount lustre-OST0001 complete [ 7609.313182] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7609.581467] Lustre: server umount lustre-OST0000 complete [ 7620.689822] Lustre: server umount lustre-MDT0000 complete [ 7625.221334] LustreError: 226886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838151 with bad export cookie 6091528048165733647 [ 7625.229170] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7625.232116] LustreError: 226886:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7625.694551] Lustre: server umount lustre-MDT0001 complete [ 7654.264756] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 09:42:58 (1787838178) [ 7663.187138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7663.549697] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7663.588522] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7663.596576] Lustre: Skipped 2 previous similar messages [ 7669.225818] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7678.852605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7679.145952] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7679.217975] Lustre: 233597:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7685.679447] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7696.311719] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7703.843034] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7712.249778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7712.590842] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7712.625917] Lustre: Permanently deactivating lustre-OST0000 [ 7712.645845] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7712.860861] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7712.865195] Lustre: Skipped 10 previous similar messages [ 7720.807713] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7722.061455] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7730.672463] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7740.492096] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7740.610870] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7747.364971] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7748.723273] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7758.421967] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7761.053872] Lustre: Permanently reactivating lustre-OST0000 [ 7761.095609] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7761.102625] Lustre: Skipped 2 previous similar messages [ 7766.584807] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7766.589486] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 44 seconds. I think it's dead, and I am evicting it. exp 00000000c93fc5dc, cur 1787838292 deadline 1787838278 last 1787838248 [ 7766.593692] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7766.628434] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7766.640027] Lustre: Skipped 1 previous similar message [ 7766.685266] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7784.721077] Lustre: server umount lustre-OST0001 complete [ 7795.616961] Lustre: server umount lustre-OST0000 complete [ 7806.470188] Lustre: server umount lustre-MDT0000 complete [ 7810.427321] LustreError: 234351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838336 with bad export cookie 6091528048165735320 [ 7810.438650] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7810.441077] LustreError: 234351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7810.994842] Lustre: server umount lustre-MDT0001 complete [ 7824.652607] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 7827.644673] Key type lgssc unregistered [ 7827.929628] LNet: 238482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7827.936333] LNetError: 238482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7827.948566] LNet: Removed LNI 192.168.204.120@tcp [ 7829.103414] Key type .llcrypt unregistered [ 7829.106440] Key type ._llcrypt unregistered [ 7855.654126] Key type ._llcrypt registered [ 7855.655836] Key type .llcrypt registered [ 7855.754687] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 7856.781068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7856.981849] alg: No test for adler32 (adler32-zlib) [ 7858.071091] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 7858.342653] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 7860.031272] Key type lgssc registered [ 7861.311725] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7871.980889] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7872.004691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7873.567477] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7878.328590] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7883.744797] LustreError: 239953:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7883.773399] LustreError: 239953:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7887.781328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7887.979255] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7887.999677] Lustre: 239942:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7888.007716] Lustre: Permanently deactivating lustre-MDT0001 [ 7888.018259] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7888.303453] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7889.104802] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7889.123893] LustreError: 239984:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7889.141754] LustreError: 239984:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7889.167754] LustreError: 239984:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 1, failed: rc = -108 [ 7893.033147] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7901.990973] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7908.593914] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7916.140463] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7916.738675] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7918.767807] LustreError: 241796:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7918.797855] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7923.169249] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7924.209325] LustreError: 241795:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7929.316461] LustreError: 241796:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7931.575248] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7933.416132] LustreError: 241794:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7933.446171] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7939.518047] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7939.757047] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7945.201386] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7945.208595] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7946.721506] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 7955.209617] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7959.432637] Lustre: Permanently reactivating lustre-MDT0001 [ 7959.457255] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7959.460450] Lustre: Skipped 2 previous similar messages [ 7965.581820] Lustre: lustre-MDT0001: Client 39957a1d-c8aa-4668-8756-8f1367470019 (at 192.168.204.20@tcp) reconnecting [ 7973.619421] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7975.855068] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7982.134590] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7982.346396] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7986.067623] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7986.445514] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7989.225477] Lustre: Permanently deactivating lustre-MDT0001 [ 7989.243078] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7989.248560] Lustre: Skipped 2 previous similar messages [ 7994.955841] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 8005.034058] Lustre: server umount lustre-MDT0000 complete [ 8006.113168] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8006.121324] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8006.135874] LustreError: 239954:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8006.154829] LustreError: 239954:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8007.665487] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8007.692449] Lustre: Skipped 2 previous similar messages [ 8009.103915] LustreError: 239935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838535 with bad export cookie 7178926817460999326 [ 8009.108340] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8009.114200] LustreError: 239935:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8009.473076] Lustre: server umount lustre-MDT0001 complete [ 8015.864079] Lustre: server umount lustre-OST0000 complete [ 8030.113719] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 09:49:14 (1787838554) [ 8041.461536] Lustre: server umount lustre-OST0001 complete [ 8049.359367] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 8062.625543] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 8104.824554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8105.032367] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8105.045299] Lustre: Skipped 2 previous similar messages [ 8105.079437] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8105.179376] Lustre: lustre-MDT0000: new disk, initializing [ 8105.323323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8105.373490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8109.702605] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8120.868597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8120.938425] Lustre: 248217:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8121.038462] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8121.046493] Lustre: Skipped 1 previous similar message [ 8121.120853] Lustre: lustre-MDT0001: new disk, initializing [ 8121.218302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8121.267835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8121.276261] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8126.603810] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8137.807336] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8144.192117] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8151.858134] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8152.166879] Lustre: lustre-OST0000: new disk, initializing [ 8152.171602] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8152.181031] Lustre: 250189:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8152.308452] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8159.332181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8159.343379] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8159.478909] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8159.766376] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8170.965209] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8173.529898] LustreError: 248224:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8183.219927] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8183.428862] Lustre: lustre-OST0001: new disk, initializing [ 8183.433425] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8183.439634] Lustre: 251304:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8183.620700] LustreError: 248224:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8183.665071] LustreError: 248225:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8189.840326] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8193.250266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8193.261497] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8193.400426] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8193.751230] LustreError: 248225:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8201.992173] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8204.788755] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8208.881524] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8208.885310] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8208.888821] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8208.892954] Lustre: Skipped 1 previous similar message [ 8208.902823] Lustre: Skipped 1 previous similar message [ 8213.767905] Lustre: server umount lustre-OST0001 complete [ 8213.988030] LustreError: 250173:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8214.026052] LustreError: 250173:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 8234.472613] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8234.488632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8234.497676] Lustre: Skipped 1 previous similar message [ 8238.723774] Lustre: server umount lustre-OST0000 complete [ 8244.192325] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8244.201219] LustreError: Skipped 1 previous similar message [ 8244.207780] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8244.222738] Lustre: Skipped 1 previous similar message [ 8244.228116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8249.824942] LustreError: 248229:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8249.836139] LustreError: 248229:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 8249.952735] Lustre: server umount lustre-MDT0000 complete [ 8254.315542] LustreError: 248209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838780 with bad export cookie 7178926817461002140 [ 8254.320040] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8254.334176] LustreError: 248209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8260.877897] Lustre: server umount lustre-MDT0001 complete [ 8272.853882] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 8275.958113] Key type lgssc unregistered [ 8276.283024] LNet: 253340:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8276.296207] LNetError: 253340:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8276.310361] LNet: Removed LNI 192.168.204.120@tcp [ 8277.573340] Key type .llcrypt unregistered [ 8277.576098] Key type ._llcrypt unregistered [ 8324.597364] Key type ._llcrypt registered [ 8324.604138] Key type .llcrypt registered [ 8324.756394] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 8326.018596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8326.069988] alg: No test for adler32 (adler32-zlib) [ 8327.472753] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 8327.762487] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 8329.448254] Key type lgssc registered [ 8330.997904] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8340.032569] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8340.058063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8341.427962] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8341.476710] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8341.686881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8346.729187] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8355.685297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8355.835070] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8355.862523] Lustre: 255550:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8356.126767] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8361.022976] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8369.033037] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8374.990070] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8381.826182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8382.099133] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8382.351264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8388.758453] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8390.001571] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8398.211463] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8455.140115] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8455.149982] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8455.163801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8455.665945] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8455.680959] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8455.695447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8458.719772] Lustre: server umount lustre-OST0000 complete [ 8463.443399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8476.975952] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8477.302031] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8484.123359] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8492.633481] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8495.662397] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8495.682833] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8495.686036] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8495.709449] Lustre: Skipped 1 previous similar message [ 8536.546516] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8536.550151] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8536.553295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8536.577049] Lustre: Skipped 1 previous similar message [ 8538.894189] Lustre: server umount lustre-OST0000 complete [ 8545.761581] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8545.772609] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8545.802608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8545.812410] Lustre: Skipped 1 previous similar message [ 8549.472482] Lustre: server umount lustre-MDT0000 complete [ 8553.169026] LustreError: 255543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839079 with bad export cookie 11908244268331921357 [ 8553.179982] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8553.185681] LustreError: 255543:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8553.711772] Lustre: server umount lustre-MDT0001 complete [ 8569.166444] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 8571.983622] Key type lgssc unregistered [ 8572.336820] LNet: 260981:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8572.343933] LNetError: 260981:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8572.360739] LNet: Removed LNI 192.168.204.120@tcp [ 8573.497265] Key type .llcrypt unregistered [ 8573.504157] Key type ._llcrypt unregistered [ 8600.671540] Key type ._llcrypt registered [ 8600.675806] Key type .llcrypt registered [ 8600.800272] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 8601.798857] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8601.832695] alg: No test for adler32 (adler32-zlib) [ 8602.826606] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 8603.121805] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 8604.815216] Key type lgssc registered [ 8605.801457] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8615.383773] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8615.416174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8616.950647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8622.417911] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8626.152307] LustreError: 262399:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8626.185857] LustreError: 262399:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8631.265686] LustreError: 262400:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8632.469589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8632.913337] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8638.604882] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8647.797438] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8654.087443] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8661.913687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8662.416761] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8663.497306] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8669.680516] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8678.178632] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8704.481405] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8704.499961] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8704.501968] Lustre: Skipped 1 previous similar message [ 8704.506517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8704.540663] LustreError: Skipped 1 previous similar message [ 8709.603925] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8709.607609] Lustre: Skipped 1 previous similar message [ 8710.484402] Lustre: server umount lustre-OST0000 complete [ 8714.729861] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8714.730943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8714.730970] Lustre: Skipped 1 previous similar message [ 8714.731490] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8714.807835] Lustre: Skipped 1 previous similar message [ 8719.841655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8719.848967] Lustre: Skipped 2 previous similar messages [ 8720.787722] Lustre: server umount lustre-MDT0000 complete [ 8724.340144] LustreError: 262381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839250 with bad export cookie 10728253555649806865 [ 8724.351866] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8724.606455] Lustre: server umount lustre-MDT0001 complete [ 8735.649843] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 8738.545627] Key type lgssc unregistered [ 8738.892820] LNet: 266720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8738.905766] LNetError: 266720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8738.923863] LNet: Removed LNI 192.168.204.120@tcp [ 8739.975533] Key type .llcrypt unregistered [ 8739.977348] Key type ._llcrypt unregistered [ 8759.701940] Key type ._llcrypt registered [ 8759.704342] Key type .llcrypt registered [ 8759.855812] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 8761.689165] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8761.740664] alg: No test for adler32 (adler32-zlib) [ 8762.942887] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 8763.343334] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 8765.111539] Key type lgssc registered [ 8766.242980] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8775.627892] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8775.647866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8777.190289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8781.842968] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8787.426700] LustreError: 267943:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8787.460841] LustreError: 267943:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8791.175874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8791.514896] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8796.724905] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8805.655437] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8813.955637] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8822.641595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8823.299819] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8830.822095] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8839.130974] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8847.941162] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8862.180898] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8862.193329] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8862.225438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8863.202680] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8863.203626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8865.333712] Lustre: server umount lustre-OST0000 complete [ 8873.454292] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8873.458506] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8873.459873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8873.459879] Lustre: Skipped 1 previous similar message [ 8873.483616] Lustre: Skipped 1 previous similar message [ 8875.240234] LustreError: 270891:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000cb360480) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 8877.371756] Lustre: server umount lustre-MDT0000 complete [ 8878.572264] LustreError: 267938:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8878.591410] LustreError: 267938:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8881.585726] LustreError: 267923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839407 with bad export cookie 11116993793677150049 [ 8881.586927] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8881.597350] LustreError: 267923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8882.098217] Lustre: server umount lustre-MDT0001 complete [ 8893.848675] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 8896.598776] Key type lgssc unregistered [ 8896.836920] LNet: 271646:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8896.844256] LNetError: 271646:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8896.857252] LNet: Removed LNI 192.168.204.120@tcp [ 8897.887379] Key type .llcrypt unregistered [ 8897.892220] Key type ._llcrypt unregistered [ 8927.831929] Key type ._llcrypt registered [ 8927.835124] Key type .llcrypt registered [ 8927.957741] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 8929.248757] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8929.475644] alg: No test for adler32 (adler32-zlib) [ 8930.716186] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 8930.927120] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 8932.567619] Key type lgssc registered [ 8934.056340] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8943.259702] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8943.272415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8944.812037] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8949.983784] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8954.345741] LustreError: 273066:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8954.379937] LustreError: 273066:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8959.460953] LustreError: 273065:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8960.272527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8960.825890] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8966.374437] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 8975.133651] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8981.639097] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8989.404743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8989.602713] LustreError: 274885:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8989.737630] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8994.809206] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8996.245482] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9006.043516] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9035.744419] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9035.750580] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9035.766175] LustreError: Skipped 1 previous similar message [ 9035.772629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9035.817870] Lustre: Skipped 1 previous similar message [ 9040.865971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9040.880608] Lustre: Skipped 2 previous similar messages [ 9041.790946] Lustre: server umount lustre-OST0000 complete [ 9048.038116] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9048.058786] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9048.102932] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9051.108174] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9051.110321] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9052.896661] Lustre: server umount lustre-MDT0000 complete [ 9056.225842] LustreError: 273066:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9056.261813] LustreError: 273066:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9057.049679] LustreError: 273046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839583 with bad export cookie 12756637357999159953 [ 9057.053394] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9057.063505] LustreError: 273046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9057.469804] Lustre: server umount lustre-MDT0001 complete [ 9069.382405] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 9072.490868] Key type lgssc unregistered [ 9072.862224] LNet: 277378:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9072.870335] LNetError: 277378:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9072.897324] LNet: Removed LNI 192.168.204.120@tcp [ 9074.096294] Key type .llcrypt unregistered [ 9074.102934] Key type ._llcrypt unregistered [ 9095.125768] Key type ._llcrypt registered [ 9095.133611] Key type .llcrypt registered [ 9095.272548] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 9096.559183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9096.844541] alg: No test for adler32 (adler32-zlib) [ 9098.016355] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 9098.359382] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 9100.087232] Key type lgssc registered [ 9101.586944] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9111.015378] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9111.043101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9112.719408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9119.036273] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9122.279977] LustreError: 278617:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9122.301941] LustreError: 278617:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9127.393970] LustreError: 278619:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9129.630754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9129.938268] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9134.767864] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9143.358869] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9150.680846] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9158.813263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9159.321209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9167.801584] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9177.496632] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9182.908226] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 9194.887462] Lustre: server umount lustre-OST0000 complete [ 9201.632805] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9201.640978] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9201.665221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9203.680413] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9203.681917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9203.705291] Lustre: Skipped 1 previous similar message [ 9208.813363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9208.832235] Lustre: Skipped 1 previous similar message [ 9209.990216] Lustre: server umount lustre-MDT0000 complete [ 9211.935149] Lustre: 277913:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787839721/real 1787839721] req@000000002d908882 x1874685721607040/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787839737 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9211.971780] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9212.003981] LustreError: 278617:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9212.033274] LustreError: 278617:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9214.047279] Lustre: 277913:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787839724/real 1787839724] req@00000000bf0f42e3 x1874685721608064/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787839740 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9214.815806] LustreError: 281568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839740 with bad export cookie 12400185126496320368 [ 9214.818808] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9214.825115] LustreError: 281568:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9215.318985] Lustre: server umount lustre-MDT0001 complete [ 9228.098479] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 9231.316473] Key type lgssc unregistered [ 9231.667749] LNet: 282324:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9231.675923] LNetError: 282324:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9231.722358] LNet: Removed LNI 192.168.204.120@tcp [ 9232.785637] Key type .llcrypt unregistered [ 9232.789093] Key type ._llcrypt unregistered [ 9245.670879] Key type ._llcrypt registered [ 9245.673553] Key type .llcrypt registered [ 9245.820829] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9266.000097] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 9283.042719] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 9284.364900] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9284.476936] alg: No test for adler32 (adler32-zlib) [ 9285.567939] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 9285.795818] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 9287.496274] Key type lgssc registered [ 9288.916804] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9328.302711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9328.323605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9329.585708] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9329.632499] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9329.722419] Lustre: lustre-MDT0000: new disk, initializing [ 9329.829315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9329.858498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9334.362362] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9346.130171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9346.245675] Lustre: 285878:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9346.294832] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9346.298479] Lustre: Skipped 1 previous similar message [ 9346.373282] Lustre: lustre-MDT0001: new disk, initializing [ 9346.497852] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9346.534199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9346.550387] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9352.322858] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9362.262513] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9367.472365] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9374.032128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9374.477512] Lustre: lustre-OST0000: new disk, initializing [ 9374.483167] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9374.491514] Lustre: 287845:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9374.710493] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9381.241756] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9381.439189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9381.452939] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9381.576192] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9392.450913] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9398.138749] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9398.368956] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9404.264384] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9404.573858] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9407.458099] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9407.472554] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9407.477981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9407.485276] Lustre: Skipped 1 previous similar message [ 9407.495904] Lustre: Skipped 1 previous similar message [ 9412.576633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9412.582882] Lustre: Skipped 1 previous similar message [ 9417.698613] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9417.721879] Lustre: Skipped 1 previous similar message [ 9421.279153] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9421.533615] Lustre: server umount lustre-OST0000 complete [ 9427.944568] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9427.949452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9427.963876] Lustre: Skipped 1 previous similar message [ 9427.978950] Lustre: Skipped 1 previous similar message [ 9432.141917] Lustre: server umount lustre-MDT0000 complete [ 9436.522813] LustreError: 285871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839962 with bad export cookie 15530203508135243609 [ 9436.534817] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9436.541828] LustreError: 285871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9436.981391] Lustre: server umount lustre-MDT0001 complete [ 9446.313664] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 10:12:50 (1787839970) [ 9456.083927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9456.918706] LustreError: 290260:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9456.964168] LustreError: 290260:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9457.076678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9462.248115] LustreError: 290260:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9462.268656] LustreError: 290260:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9463.035422] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9466.343469] LustreError: 290261:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9471.477658] LustreError: 290260:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9476.068878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9476.576467] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9477.029960] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9483.323908] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9495.645687] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9503.275758] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9511.166953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9511.440422] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9518.014165] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9528.023050] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9542.130622] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9542.134864] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9542.146157] LustreError: Skipped 2 previous similar messages [ 9542.156819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9542.171872] Lustre: Skipped 1 previous similar message [ 9542.196396] Lustre: Skipped 1 previous similar message [ 9546.311798] Lustre: server umount lustre-OST0000 complete [ 9552.360482] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9557.727964] Lustre: server umount lustre-MDT0000 complete [ 9562.618664] LustreError: 290260:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9562.649843] LustreError: 290260:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9562.743662] LustreError: 290240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840088 with bad export cookie 15530203508135244484 [ 9562.748034] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9562.757280] LustreError: 290240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9563.083840] Lustre: server umount lustre-MDT0001 complete [ 9577.067020] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [ 9580.554930] Key type lgssc unregistered [ 9580.949731] LNet: 293802:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9580.960026] LNetError: 293802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9580.986908] LNet: Removed LNI 192.168.204.120@tcp [ 9582.010190] Key type .llcrypt unregistered [ 9582.024154] Key type ._llcrypt unregistered [ 9613.593959] Key type ._llcrypt registered [ 9613.600119] Key type .llcrypt registered [ 9613.758316] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 9630.700783] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 9631.551407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9631.761318] alg: No test for adler32 (adler32-zlib) [ 9633.054979] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [ 9633.268496] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [ 9634.904401] Key type lgssc registered [ 9636.561383] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9689.650987] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9689.700692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9691.066169] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9691.115116] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9691.280789] Lustre: lustre-MDT0000: new disk, initializing [ 9691.431696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9691.478308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9697.534258] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9709.684427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9709.990230] Lustre: 297630:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9710.122879] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9710.130448] Lustre: Skipped 1 previous similar message [ 9710.283744] Lustre: lustre-MDT0001: new disk, initializing [ 9710.442817] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9710.489734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9710.499171] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9716.185310] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9726.287861] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9732.820231] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9740.486093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9740.753824] Lustre: lustre-OST2710: new disk, initializing [ 9740.757145] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9740.764111] Lustre: 299602:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9740.838241] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9747.527531] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9751.178661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9751.191852] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9751.329462] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9757.372711] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9764.715169] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9764.938326] Lustre: lustre-OST03e8: new disk, initializing [ 9764.944618] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9764.952752] Lustre: 300666:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9765.066373] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9772.080558] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9774.185639] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9774.201422] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9774.382762] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9780.004431] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9788.398941] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9788.641711] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9793.646956] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9793.941331] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9799.162689] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9799.388816] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9805.410905] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9805.637809] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9812.718024] Lustre: server umount lustre-MDT0000 complete [ 9812.962899] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9812.965392] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9812.986424] LustreError: 299197:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9813.001683] LustreError: 299197:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9815.525918] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9815.528442] LustreError: 300275:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9815.543253] Lustre: Skipped 2 previous similar messages [ 9815.576684] LustreError: 300275:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9815.826289] LustreError: 297624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840341 with bad export cookie 12900979575466841204 [ 9815.836101] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9815.837518] LustreError: 297624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9816.226396] Lustre: server umount lustre-MDT0001 complete [ 9830.003194] Lustre: server umount lustre-OST2710 complete [ 9844.613990] Lustre: server umount lustre-OST03e8 complete [ 9862.763510] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [ 9874.707323] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [ 9911.089479] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9911.306795] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9911.335804] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9911.407796] Lustre: lustre-MDT0000: new disk, initializing [ 9911.485122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9911.501336] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9915.848644] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9928.916884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9929.218458] Lustre: 305917:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9929.443512] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9929.447023] Lustre: Skipped 1 previous similar message [ 9929.571803] Lustre: lustre-MDT0001: new disk, initializing [ 9929.746105] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9929.828882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9929.845620] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9935.859932] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9946.869839] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9953.826732] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9961.348589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9961.666638] Lustre: lustre-OST0000: new disk, initializing [ 9961.673942] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9961.692980] Lustre: 307886:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9961.817558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9968.384406] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [ 9971.795640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9971.811860] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9971.919692] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9978.013317] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9982.561018] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9982.776811] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9987.283468] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9987.552096] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9992.677678] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9992.681926] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9992.698240] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9992.724617] Lustre: Skipped 1 previous similar message [ 9996.024898] Lustre: server umount lustre-OST0000 complete [10001.376169] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10001.379031] LustreError: Skipped 1 previous similar message [10001.382676] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10001.402605] Lustre: Skipped 1 previous similar message [10001.409861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10002.913708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10002.924726] Lustre: Skipped 1 previous similar message [10006.322803] Lustre: server umount lustre-MDT0000 complete [10008.035741] LustreError: 305925:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10008.074172] LustreError: 305925:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10010.031754] LustreError: 305910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840536 with bad export cookie 12900979575466842884 [10010.036755] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10010.054598] LustreError: 305910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10010.418196] Lustre: server umount lustre-MDT0001 complete [10020.637875] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 10:22:24 (1787840544) [10022.521899] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [10024.825256] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 10:22:29 (1787840549) [10029.044560] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [10059.746521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10060.055933] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10060.087329] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10060.288857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10065.294496] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10074.923064] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10075.044959] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10075.064120] Lustre: 311744:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10080.284538] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10103.086176] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10103.255248] Lustre: Denying initial registration attempt from nid 192.168.204.120@tcp, specified as failover [10103.262685] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [10103.274623] LustreError: 313200:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [10103.293909] LustreError: 313200:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10103.301112] LustreError: 313200:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10103.368757] Lustre: server umount lustre-OST0000 complete [10103.372307] LustreError: 313200:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [10111.467711] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10111.475108] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10111.493102] Lustre: Skipped 1 previous similar message [10111.510653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10117.095759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10117.100981] Lustre: Skipped 2 previous similar messages [10124.258223] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10124.512409] Lustre: server umount lustre-MDT0000 complete [10127.328096] LustreError: 311758:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10127.346791] LustreError: 311758:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10128.439628] LustreError: 311739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840654 with bad export cookie 12900979575466843766 [10128.448107] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10128.454301] LustreError: 311739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10128.806369] Lustre: server umount lustre-MDT0001 complete [10137.726633] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [10140.918772] Key type lgssc unregistered [10141.312124] LNet: 314326:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10141.325623] LNetError: 314326:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10141.346187] LNet: Removed LNI 192.168.204.120@tcp [10142.439756] Key type .llcrypt unregistered [10142.445492] Key type ._llcrypt unregistered [10156.431413] Key type ._llcrypt registered [10156.433435] Key type .llcrypt registered [10156.591214] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [10157.863924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10157.899878] alg: No test for adler32 (adler32-zlib) [10159.089411] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [10159.385876] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [10161.047196] Key type lgssc registered [10162.367821] Lustre: Echo OBD driver; http://www.lustre.org/ [10214.272921] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [10225.895622] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10225.959194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10227.540465] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10227.569443] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10227.829846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10233.595978] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10244.159879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10244.297414] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10244.335606] Lustre: 317093:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10244.605235] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10249.444811] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10257.673141] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10263.656201] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10285.758135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10286.088171] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10286.340738] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10296.897848] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10305.722977] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10308.067385] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10308.070590] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10308.080371] LustreError: Skipped 1 previous similar message [10308.084957] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10308.093081] Lustre: Skipped 1 previous similar message [10308.130913] Lustre: Skipped 1 previous similar message [10313.203288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10313.213192] Lustre: Skipped 1 previous similar message [10318.313683] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10318.318801] Lustre: Skipped 1 previous similar message [10321.887187] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10322.139649] Lustre: server umount lustre-OST0000 complete [10328.544794] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10328.552268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10328.570069] Lustre: Skipped 1 previous similar message [10328.581659] Lustre: Skipped 1 previous similar message [10333.666034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10333.674061] Lustre: Skipped 1 previous similar message [10341.343153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10341.671684] Lustre: server umount lustre-MDT0000 complete [10346.324958] LustreError: 317086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840872 with bad export cookie 9577421975264867828 [10346.327281] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10346.350392] LustreError: 317086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10346.840397] Lustre: server umount lustre-MDT0001 complete [10359.693631] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [10363.406617] Key type lgssc unregistered [10363.732080] LNet: 320674:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10363.744331] LNetError: 320674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10363.767486] LNet: Removed LNI 192.168.204.120@tcp [10365.038134] Key type .llcrypt unregistered [10365.040038] Key type ._llcrypt unregistered [10394.040328] Key type ._llcrypt registered [10394.041686] Key type .llcrypt registered [10394.164410] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [10395.660100] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10395.728924] alg: No test for adler32 (adler32-zlib) [10397.125712] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [10397.491289] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [10399.320175] Key type lgssc registered [10401.169752] Lustre: Echo OBD driver; http://www.lustre.org/ [10410.252597] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10410.295563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10411.690610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10416.549821] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10421.226978] LustreError: 322095:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10421.245525] LustreError: 322095:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10426.341548] LustreError: 322094:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10427.986134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10428.479314] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10433.654197] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10443.577437] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10451.339767] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10461.871093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10462.375161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10470.034601] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10480.319259] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10494.949565] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10494.957304] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10494.975599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10496.391206] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.20@tcp (stopping) [10497.505182] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10497.511354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10497.529945] Lustre: Skipped 1 previous similar message [10497.535379] Lustre: Skipped 1 previous similar message [10501.080856] Lustre: server umount lustre-MDT0000 complete [10501.474896] LustreError: 324191:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10505.213041] LustreError: 322075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841031 with bad export cookie 16315389499001855204 [10505.221944] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10506.604288] LustreError: 322089:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10506.624944] LustreError: 322089:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10506.648079] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.20@tcp (stopping) [10506.654408] Lustre: Skipped 1 previous similar message [10507.746243] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10510.754654] LustreError: 322090:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10510.778098] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10510.778880] LustreError: 322090:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10510.797654] Lustre: Skipped 1 previous similar message [10511.626644] Lustre: server umount lustre-MDT0001 complete [10519.569450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10530.095398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10530.271786] LustreError: 321404:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000002a3b9590 x1874687083719168/t0(0) o250->MGC192.168.204.120@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10530.779867] LustreError: 323928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10530.910226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10531.028266] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10537.390832] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10540.215490] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10547.041561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10547.440383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10552.683378] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10552.822291] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10562.875962] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10564.898381] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10572.211647] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10574.093820] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10581.986758] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10581.999144] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10582.015367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10583.520994] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10586.536799] Lustre: server umount lustre-OST0000 complete [10593.761505] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10593.765420] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10593.770228] Lustre: Skipped 2 previous similar messages [10596.838931] Lustre: server umount lustre-MDT0000 complete [10598.885722] LustreError: 325716:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10598.915462] LustreError: 325716:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [10601.198114] LustreError: 322074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841127 with bad export cookie 16315389499001862743 [10601.212722] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10601.228300] LustreError: 322074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [10601.570677] Lustre: server umount lustre-MDT0001 complete [10613.674363] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [10616.195787] Key type lgssc unregistered [10616.433475] LNet: 328390:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10616.437217] LNetError: 328390:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10616.452072] LNet: Removed LNI 192.168.204.120@tcp [10617.362679] Key type .llcrypt unregistered [10617.367458] Key type ._llcrypt unregistered [10647.756307] Key type ._llcrypt registered [10647.758097] Key type .llcrypt registered [10647.883382] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [10648.995628] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10649.067852] alg: No test for adler32 (adler32-zlib) [10650.314936] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [10650.669450] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [10652.424177] Key type lgssc registered [10653.913222] Lustre: Echo OBD driver; http://www.lustre.org/ [10664.039921] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10664.072877] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10665.466139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10670.784486] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10675.684738] LustreError: 329807:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10675.714793] LustreError: 329807:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10679.966712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10680.411135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10685.744139] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10696.145932] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10705.152920] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10709.392410] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10709.411831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10709.438036] Lustre: Skipped 1 previous similar message [10714.595782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10714.605168] Lustre: Skipped 1 previous similar message [10715.827860] Lustre: server umount lustre-MDT0000 complete [10719.720720] LustreError: 329807:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10719.746310] LustreError: 329807:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10720.868726] LustreError: 329788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841246 with bad export cookie 9033041937333397939 [10720.894178] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10721.250509] Lustre: server umount lustre-MDT0001 complete [10737.679579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10738.166616] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10738.226656] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10738.472149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10743.971490] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10755.587615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10755.808837] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10755.891302] Lustre: 332399:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10756.525766] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10762.635881] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10772.477745] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10778.864173] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10789.618464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10789.833804] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10789.841490] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10789.847746] LustreError: 334284:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [10789.852527] LustreError: 334284:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10789.855240] LustreError: 334284:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10789.929624] Lustre: server umount lustre-OST0000 complete [10789.933340] LustreError: 334284:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10802.240533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10802.654365] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10802.981476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10809.278374] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10812.007440] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10818.265883] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10830.172554] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10830.335519] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10830.530374] Lustre: lustre-OST0001: new disk, initializing [10830.550779] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10830.565554] Lustre: 335827:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10830.694719] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10838.636431] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [10840.317077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10840.329280] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10840.534073] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10848.648771] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10855.916682] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10855.921213] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10855.923339] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10855.926433] LustreError: Skipped 1 previous similar message [10855.956565] Lustre: Skipped 2 previous similar messages [10861.033269] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10861.046550] Lustre: Skipped 2 previous similar messages [10866.143878] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10866.157746] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10866.343815] Lustre: server umount lustre-OST0001 complete [10871.265866] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10871.281676] Lustre: Skipped 1 previous similar message [10871.282899] LustreError: 334752:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10871.304134] LustreError: 334752:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10876.399043] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10876.409026] LustreError: 336243:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10876.416835] Lustre: Skipped 4 previous similar messages [10876.459277] LustreError: 336243:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10877.557136] Lustre: server umount lustre-OST0000 complete [10884.579536] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10884.586494] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10888.483095] Lustre: server umount lustre-MDT0000 complete [10891.746860] LustreError: 336632:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10891.776382] LustreError: 336632:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10892.270944] LustreError: 334767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841418 with bad export cookie 9033041937333398492 [10892.275702] LustreError: 334767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10892.278394] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10892.769780] Lustre: server umount lustre-MDT0001 complete [10905.617578] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [10908.654483] Key type lgssc unregistered [10908.941960] LNet: 337792:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10908.950683] LNetError: 337792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10908.963218] LNet: Removed LNI 192.168.204.120@tcp [10909.703327] Key type .llcrypt unregistered [10909.711292] Key type ._llcrypt unregistered [10986.499518] Key type ._llcrypt registered [10986.501838] Key type .llcrypt registered [10986.563199] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [11002.326852] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [11003.904427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11003.927369] alg: No test for adler32 (adler32-zlib) [11005.008987] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [11005.314128] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [11007.048343] Key type lgssc registered [11008.991441] Lustre: Echo OBD driver; http://www.lustre.org/ [11053.427608] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11053.454986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11054.685367] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11054.729581] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11054.794596] Lustre: lustre-MDT0000: new disk, initializing [11054.877945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11054.897091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11059.426485] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11071.290483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11071.434260] Lustre: 342729:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11071.515322] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11071.521478] Lustre: Skipped 1 previous similar message [11071.648954] Lustre: lustre-MDT0001: new disk, initializing [11071.789825] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11071.849924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11071.875544] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11076.221893] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11087.497761] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11094.196382] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11102.110801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11102.442763] Lustre: lustre-OST0000: new disk, initializing [11102.449913] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11102.464660] Lustre: 344696:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11102.639812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11109.626992] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11110.470496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11110.483480] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11110.687129] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11120.368701] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11126.348127] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11126.666488] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11132.156842] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11132.518994] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11136.482174] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11136.489993] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11136.517117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11136.550608] Lustre: Skipped 1 previous similar message [11136.579162] Lustre: Skipped 1 previous similar message [11141.012658] Lustre: server umount lustre-OST0000 complete [11146.722376] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11146.729297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11146.745724] Lustre: Skipped 1 previous similar message [11146.768649] Lustre: Skipped 1 previous similar message [11152.015580] Lustre: server umount lustre-MDT0000 complete [11156.609859] LustreError: 342718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841682 with bad export cookie 11060342920499052687 [11156.621151] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11156.632181] LustreError: 342718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11157.273725] Lustre: server umount lustre-MDT0001 complete [11168.346309] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 10:41:32 (1787841692) [11177.444205] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 10:41:41 (1787841701) [11187.345448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11187.810938] LustreError: 347454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11187.828620] LustreError: 347454:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11187.884368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11193.314660] LustreError: 347455:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11193.405747] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11197.410623] LustreError: 347454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11202.529805] LustreError: 347454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11203.477194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11204.002985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11209.603141] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11218.527530] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11226.369686] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11234.588712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11235.174912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11242.715395] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11251.384511] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11268.927840] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [11271.115719] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [11273.098707] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [11275.000816] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [11276.972028] Lustre: Failing over lustre-MDT0000 [11277.246348] Lustre: server umount lustre-MDT0000 complete [11277.687624] LustreError: 347449:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11278.308505] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11278.329250] Lustre: Skipped 1 previous similar message [11287.912690] LustreError: 347449:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11287.936396] LustreError: 347449:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [11294.687098] Lustre: 340952:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787841804/real 1787841804] req@000000000ab1b739 x1874687721285888/t0(0) o400->MGC192.168.204.120@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787841820 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11294.718467] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11296.619956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11304.295738] LustreError: 349513:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11304.328678] LustreError: 349513:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [11304.954169] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x997e3b49365a481b [11304.972469] Lustre: MGC192.168.204.120@tcp: Connection restored to 0@lo (at 0@lo) [11305.285681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11305.359037] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11309.441825] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11309.946567] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11310.587799] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11310.648065] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [11310.724766] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11320.988456] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11322.790493] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11326.511669] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [11328.918414] Lustre: Failing over lustre-MDT0000 [11329.492555] Lustre: server umount lustre-MDT0000 complete [11331.045809] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11331.048556] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11331.059405] Lustre: Skipped 2 previous similar messages [11340.130590] LustreError: 349513:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11340.155304] LustreError: 349513:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [11347.424922] Lustre: 340950:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787841857/real 1787841857] req@000000009b406a7e x1874687721307136/t0(0) o400->MGC192.168.204.120@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787841873 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11347.461514] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11349.775580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11357.675501] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x997e3b49365a4b2b [11357.694833] Lustre: MGC192.168.204.120@tcp: Connection restored to 0@lo (at 0@lo) [11357.698871] Lustre: Skipped 2 previous similar messages [11357.966647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11358.019215] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11360.612510] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11363.332917] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11363.403255] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [11363.439116] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11363.499578] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11373.688623] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11375.462811] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11383.780708] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11383.793480] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11383.809912] Lustre: Skipped 1 previous similar message [11383.815738] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11383.841518] Lustre: Skipped 2 previous similar messages [11387.930626] Lustre: server umount lustre-MDT0000 complete [11392.309810] LustreError: 347437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841918 with bad export cookie 11060342920499055403 [11392.325669] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11392.328785] LustreError: 347437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11392.759651] Lustre: server umount lustre-MDT0001 complete [11407.683367] Lustre: server umount lustre-OST0000 complete [11424.328368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11424.707242] LustreError: 353820:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11424.729956] LustreError: 353820:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 21 previous similar messages [11424.798945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11429.563966] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11439.703582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11445.350764] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11453.977603] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11460.659050] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11468.028760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11470.518635] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11475.726888] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11484.750740] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11496.419314] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11496.421269] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11496.424769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11496.432686] LustreError: Skipped 1 previous similar message [11496.490813] Lustre: Skipped 1 previous similar message [11501.538765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11501.554839] Lustre: Skipped 1 previous similar message [11502.581871] Lustre: server umount lustre-OST0000 complete [11511.776223] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11511.784550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11511.793393] Lustre: Skipped 1 previous similar message [11513.318827] Lustre: server umount lustre-MDT0000 complete [11517.428792] LustreError: 353800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842043 with bad export cookie 11060342920499056145 [11517.440040] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11517.454896] LustreError: 353800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11517.917462] Lustre: server umount lustre-MDT0001 complete [11529.969900] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [11533.042538] Key type lgssc unregistered [11533.418878] LNet: 357355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11533.432096] LNetError: 357355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11533.456083] LNet: Removed LNI 192.168.204.120@tcp [11534.426828] Key type .llcrypt unregistered [11534.430506] Key type ._llcrypt unregistered [11562.994278] Key type ._llcrypt registered [11562.995885] Key type .llcrypt registered [11563.067972] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [11563.763484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11563.816981] alg: No test for adler32 (adler32-zlib) [11564.850730] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [11565.049881] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [11566.735525] Key type lgssc registered [11567.640882] Lustre: Echo OBD driver; http://www.lustre.org/ [11575.485537] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11575.523327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11577.046943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11581.803958] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11586.535032] LustreError: 358873:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11586.559388] LustreError: 358873:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11591.329837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11591.705237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11596.550657] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11604.240144] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11610.438979] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11617.633536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11617.962924] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11621.061716] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11624.637475] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11632.319488] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11647.969598] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11647.973573] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11647.990798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11648.230868] Lustre: server umount lustre-MDT0000 complete [11651.974984] LustreError: 358854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842177 with bad export cookie 1853877245073002492 [11651.991299] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11651.993468] LustreError: 358854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11652.064924] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11652.067915] LustreError: 360715:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11652.085749] Lustre: Skipped 1 previous similar message [11652.106653] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11652.385777] Lustre: server umount lustre-MDT0001 complete [11656.358768] Lustre: server umount lustre-OST0000 complete [11670.938856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11671.403766] LustreError: 362608:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11671.492798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11676.271835] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11676.642163] LustreError: 362609:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11681.765297] LustreError: 362608:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11685.671786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11686.087067] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11691.393440] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11700.576659] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11706.272761] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11714.046676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11714.600952] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11716.658709] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11722.430153] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11731.527809] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11746.283545] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11746.286307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11746.305542] Lustre: Skipped 2 previous similar messages [11746.315745] Lustre: Skipped 1 previous similar message [11749.446802] Lustre: server umount lustre-MDT0000 complete [11751.396730] LustreError: 362609:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11751.436291] LustreError: 362609:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11754.043910] LustreError: 362589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842280 with bad export cookie 1853877245073003612 [11754.045832] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11754.058258] LustreError: 362589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11754.445596] Lustre: server umount lustre-MDT0001 complete [11769.167507] Lustre: server umount lustre-OST0000 complete [11786.008197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11786.682309] LustreError: 366521:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11786.780411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11792.108446] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11802.828189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11807.644052] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11814.754699] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11821.122518] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11828.236639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11828.561507] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11828.567087] Lustre: Skipped 1 previous similar message [11831.660519] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11834.665133] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [11842.453800] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11849.743928] Lustre: server umount lustre-OST0000 complete [11854.311515] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11854.328306] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11854.344116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11854.348165] Lustre: Skipped 1 previous similar message [11862.497439] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11862.510221] Lustre: Skipped 3 previous similar messages [11863.630637] Lustre: server umount lustre-MDT0000 complete [11866.532359] Lustre: 358185:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787842376/real 1787842376] req@000000008bf6fdd8 x1874688308503424/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787842392 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11866.557156] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11866.573601] Lustre: Skipped 1 previous similar message [11866.577910] LustreError: 366522:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11866.589703] LustreError: 366522:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11867.676067] LustreError: 366502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842393 with bad export cookie 1853877245073004627 [11867.677639] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11867.684493] LustreError: 366502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11868.106746] Lustre: server umount lustre-MDT0001 complete [11879.460600] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [11881.962355] Key type lgssc unregistered [11882.295219] LNet: 370055:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11882.312121] LNetError: 370055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11883.371208] LNet: Removed LNI 192.168.204.120@tcp [11884.214547] Key type .llcrypt unregistered [11884.218634] Key type ._llcrypt unregistered [11911.268582] Key type ._llcrypt registered [11911.275323] Key type .llcrypt registered [11911.385781] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [11912.293462] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11912.358147] alg: No test for adler32 (adler32-zlib) [11913.445587] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [11913.771966] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [11915.465929] Key type lgssc registered [11916.801287] Lustre: Echo OBD driver; http://www.lustre.org/ [11925.972986] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11925.998257] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11926.008093] LustreError: 371498:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11926.019464] LustreError: 371498:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11926.024217] LustreError: 371498:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11926.028677] LustreError: 371498:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11926.032885] LustreError: 371498:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11934.275248] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11934.283780] LustreError: 371895:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11934.295364] LustreError: 371895:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11934.300632] LustreError: 371895:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11934.306358] LustreError: 371895:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11934.313080] LustreError: 371895:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11957.140043] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [11960.256437] Key type lgssc unregistered [11960.624898] LNet: 372903:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11960.638813] LNetError: 372903:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11960.662843] LNet: Removed LNI 192.168.204.120@tcp [11961.888422] Key type .llcrypt unregistered [11961.890657] Key type ._llcrypt unregistered [11982.581139] Key type ._llcrypt registered [11982.584939] Key type .llcrypt registered [11982.696900] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [12000.273928] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [12001.204666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12001.460457] alg: No test for adler32 (adler32-zlib) [12002.563550] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [12002.861247] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [12004.599591] Key type lgssc registered [12006.133445] Lustre: Echo OBD driver; http://www.lustre.org/ [12044.280445] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12044.310742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12045.594751] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12045.612363] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12045.675247] Lustre: lustre-MDT0000: new disk, initializing [12045.724231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12045.747200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12050.124483] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12062.341862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12062.433204] Lustre: 376062:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12062.480373] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12062.486557] Lustre: Skipped 1 previous similar message [12062.563290] Lustre: lustre-MDT0001: new disk, initializing [12062.666786] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12062.706966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12062.724291] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12067.807683] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12077.779956] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12085.246774] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12092.794054] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12093.333856] Lustre: lustre-OST0000: new disk, initializing [12093.339984] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12093.350466] Lustre: 378031:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12093.446427] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12100.491791] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12101.225620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12101.232881] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12101.375250] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12111.772282] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12116.424647] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12116.659701] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12121.507662] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12121.749504] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12127.203469] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12127.212546] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12127.216593] LustreError: Skipped 1 previous similar message [12127.231082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12127.249174] Lustre: Skipped 1 previous similar message [12129.743499] Lustre: server umount lustre-OST0000 complete [12134.378783] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12134.393995] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12134.407893] Lustre: Skipped 1 previous similar message [12134.416950] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12137.443722] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12137.447196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12137.473375] Lustre: Skipped 1 previous similar message [12139.716956] Lustre: server umount lustre-MDT0000 complete [12144.114277] LustreError: 376055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842670 with bad export cookie 14271028793363063241 [12144.122763] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12144.136086] LustreError: 376055:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12144.896454] Lustre: server umount lustre-MDT0001 complete [12155.803094] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [12158.714592] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 10:58:02 (1787842682) [12167.072727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12167.682307] LustreError: 380491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12167.700842] LustreError: 380491:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12167.792063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12172.772466] LustreError: 380492:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12172.921245] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12177.890293] LustreError: 380491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12181.859035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12182.196540] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12186.706173] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12195.148961] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12201.086129] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12208.683560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12209.126118] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12215.591095] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12223.971817] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12231.874851] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12232.112397] Lustre: lustre-OST0001: new disk, initializing [12232.117803] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12232.127852] Lustre: 383327:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12232.208045] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12239.861957] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12242.044642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12242.069617] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12242.137252] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12250.356281] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12257.156780] Lustre: lustre-OST0001: Not available for connect from 192.168.204.20@tcp (stopping) [12257.764462] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12257.779146] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12261.046939] Lustre: server umount lustre-OST0001 complete [12262.245201] LustreError: 383537:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12268.005288] LustreError: 383537:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12268.032880] LustreError: 383537:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12278.247476] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12278.247863] LustreError: 382315:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12278.252456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12278.252466] Lustre: Skipped 2 previous similar messages [12278.258844] Lustre: Skipped 2 previous similar messages [12278.282053] LustreError: 382315:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12288.499441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12288.505388] Lustre: Skipped 4 previous similar messages [12292.063136] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12292.175193] Lustre: server umount lustre-OST0000 complete [12298.721857] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12298.743430] Lustre: Skipped 1 previous similar message [12301.895744] Lustre: server umount lustre-MDT0000 complete [12303.849103] LustreError: 380491:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12303.869229] LustreError: 380491:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12305.686697] LustreError: 381228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842831 with bad export cookie 14271028793363064123 [12305.698901] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12305.700955] LustreError: 381228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12312.253028] Lustre: server umount lustre-MDT0001 complete [12324.047969] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [12326.924928] Key type lgssc unregistered [12327.222096] LNet: 385329:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12327.239499] LNetError: 385329:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12327.252476] LNet: Removed LNI 192.168.204.120@tcp [12327.997935] Key type .llcrypt unregistered [12328.003148] Key type ._llcrypt unregistered [12382.667085] Key type ._llcrypt registered [12382.668973] Key type .llcrypt registered [12382.769723] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [12384.124449] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12384.166650] alg: No test for adler32 (adler32-zlib) [12385.312941] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [12385.672043] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [12387.431200] Key type lgssc registered [12388.593468] Lustre: Echo OBD driver; http://www.lustre.org/ [12396.903383] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12396.927493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12398.199360] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12398.219348] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12398.409345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12403.769816] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12413.467627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12413.541221] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12413.582321] Lustre: 388140:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12413.914793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12419.087439] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12428.092755] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12434.478278] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12439.522627] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12439.531167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12439.535725] Lustre: Skipped 1 previous similar message [12439.537702] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12439.553275] Lustre: Skipped 1 previous similar message [12442.759876] Lustre: server umount lustre-MDT0000 complete [12447.207338] LustreError: 388133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842973 with bad export cookie 10725038846129867374 [12447.232605] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12447.241331] LustreError: 388133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12447.519306] Lustre: server umount lustre-MDT0001 complete [12460.151167] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 11:03:04 (1787842984) [12469.028419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12469.626369] LustreError: 390653:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12469.654239] LustreError: 390653:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12469.747302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12474.851118] LustreError: 390654:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12475.041280] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12479.982860] LustreError: 390653:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12484.056392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12484.415401] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12489.792600] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12498.584623] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12505.278156] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12512.509309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12512.806724] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12513.057631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12520.722471] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12529.039564] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12537.094857] Lustre: Permanently deactivating lustre-OST0000 [12537.111950] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12546.618574] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12548.764465] LustreError: 393688:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12548.772788] LustreError: 393688:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12548.781782] LustreError: 393688:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12552.374565] LustreError: 393736:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12552.386462] LustreError: 393736:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12560.425959] Lustre: server umount lustre-OST0000 complete [12563.384697] LustreError: 393987:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12563.388042] LustreError: 393987:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12565.475380] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12565.477364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12565.485691] Lustre: Skipped 1 previous similar message [12565.492344] Lustre: Skipped 1 previous similar message [12570.599497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12570.618504] Lustre: Skipped 1 previous similar message [12575.716238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12575.730544] Lustre: Skipped 1 previous similar message [12575.809630] Lustre: server umount lustre-MDT0000 complete [12579.651488] LustreError: 393344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843105 with bad export cookie 10725038846129867955 [12579.655506] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12579.667093] LustreError: 393344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12589.217388] Lustre: server umount lustre-MDT0001 complete [12599.204268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12604.383595] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12607.586948] LustreError: 395148:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12615.590393] Lustre: 395488:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.120@tcp [12618.481859] Lustre: server umount MGS complete [12634.051173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12634.710381] LustreError: 396214:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12634.773706] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12634.781730] Lustre: Skipped 1 previous similar message [12634.870799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12640.226320] LustreError: 396215:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12642.097465] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12650.479617] LustreError: 396215:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12650.511911] LustreError: 396215:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12652.672658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12653.450129] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12653.633685] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12659.242863] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12669.497887] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12677.306526] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12679.635543] Lustre: Permanently reactivating lustre-OST0000 [12679.679078] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12679.684437] Lustre: Skipped 2 previous similar messages [12707.300156] LustreError: 397954:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12717.632354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12717.891814] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12726.192921] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12735.689205] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12742.626134] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12742.637751] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12742.655493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12743.589775] Lustre: server umount lustre-OST0000 complete [12748.770219] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12748.788258] Lustre: Skipped 1 previous similar message [12753.905677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12753.914455] Lustre: Skipped 3 previous similar messages [12758.110188] Lustre: server umount lustre-MDT0000 complete [12759.008913] LustreError: 396214:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12759.024602] LustreError: 396214:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12761.879128] LustreError: 396195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843287 with bad export cookie 10725038846129869005 [12761.891202] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12761.897774] LustreError: 396195:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12762.352733] Lustre: server umount lustre-MDT0001 complete [12775.201760] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [12779.484409] Key type lgssc unregistered [12779.944514] LNet: 400534:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12779.961355] LNetError: 400534:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12779.998337] LNet: Removed LNI 192.168.204.120@tcp [12781.346600] Key type .llcrypt unregistered [12781.349483] Key type ._llcrypt unregistered [12803.229958] Key type ._llcrypt registered [12803.232806] Key type .llcrypt registered [12803.415745] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [12821.090140] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [12822.697729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12822.717092] alg: No test for adler32 (adler32-zlib) [12823.803828] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [12824.089803] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [12825.815186] Key type lgssc registered [12827.008305] Lustre: Echo OBD driver; http://www.lustre.org/ [12868.600869] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12868.626702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12869.887130] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12869.948610] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12870.124660] Lustre: lustre-MDT0000: new disk, initializing [12870.267951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12870.325319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12877.170140] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12889.135557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12889.290186] Lustre: 403696:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12889.363834] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12889.367456] Lustre: Skipped 1 previous similar message [12889.442733] Lustre: lustre-MDT0001: new disk, initializing [12889.593532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12889.628944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12889.673794] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12895.462621] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12905.956664] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12912.860086] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12921.015178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12921.377930] Lustre: lustre-OST0000: new disk, initializing [12921.387475] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12921.403656] Lustre: 405668:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12921.532853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12929.715604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12929.747528] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12929.898986] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12931.268246] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [12944.078954] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12949.368064] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12949.688751] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12955.783795] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12955.941768] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12960.741887] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12960.765821] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12960.773662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12964.148567] Lustre: server umount lustre-OST0000 complete [12970.981968] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12971.002612] Lustre: Skipped 1 previous similar message [12971.015654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12971.029113] Lustre: Skipped 2 previous similar messages [12976.096506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12976.107088] Lustre: Skipped 1 previous similar message [12980.199909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12980.206724] Lustre: Skipped 1 previous similar message [12983.264062] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12983.465456] Lustre: server umount lustre-MDT0000 complete [12988.473198] LustreError: 403689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843514 with bad export cookie 14778271798572349358 [12988.477774] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12988.489588] LustreError: 403689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12988.831401] Lustre: server umount lustre-MDT0001 complete [12999.618887] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [13002.100829] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 11:12:06 (1787843526) [13009.801917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13010.338395] LustreError: 408131:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13010.362823] LustreError: 408131:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13010.451320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13015.196135] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [13015.522895] LustreError: 408132:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13020.643323] LustreError: 408131:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13024.569747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13025.027326] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13029.935451] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [13039.369619] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13046.324891] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13055.097158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13055.536578] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13064.311562] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [13075.421439] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13078.359614] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [13081.166250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [13090.287155] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13090.299944] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13090.309630] Lustre: Skipped 1 previous similar message [13090.310741] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13090.327177] LustreError: Skipped 1 previous similar message [13095.224612] Lustre: server umount lustre-OST0000 complete [13100.512773] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13100.537553] Lustre: Skipped 1 previous similar message [13100.545606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13100.558807] Lustre: Skipped 2 previous similar messages [13106.537729] Lustre: server umount lustre-MDT0000 complete [13110.753295] LustreError: 409574:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13110.782149] LustreError: 409574:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13111.379804] LustreError: 408112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843637 with bad export cookie 14778271798572350240 [13111.386900] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13111.396327] LustreError: 408112:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13111.906867] Lustre: server umount lustre-MDT0001 complete [13125.158356] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [13128.771609] Key type lgssc unregistered [13129.110701] LNet: 411862:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13129.127389] LNetError: 411862:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13129.152600] LNet: Removed LNI 192.168.204.120@tcp [13130.354452] Key type .llcrypt unregistered [13130.356799] Key type ._llcrypt unregistered [13159.087569] Key type ._llcrypt registered [13159.091138] Key type .llcrypt registered [13159.269573] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [13159.685559] Key type .llcrypt unregistered [13159.690110] Key type ._llcrypt unregistered [13180.532376] Key type ._llcrypt registered [13180.536559] Key type .llcrypt registered [13180.695837] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [13182.199275] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13182.290845] alg: No test for adler32 (adler32-zlib) [13183.871265] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [13184.331823] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [13186.145804] Key type lgssc registered [13187.949228] Lustre: Echo OBD driver; http://www.lustre.org/ [13196.688925] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13196.724050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13198.341559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13204.370395] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [13208.546489] LustreError: 414079:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13208.571755] LustreError: 414079:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13213.664651] LustreError: 414080:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13214.911805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13215.213182] LustreError: 414079:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13215.222635] LustreError: 414079:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13215.242912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13220.321594] LustreError: 414080:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13220.345922] LustreError: 414080:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13222.894578] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [13223.440797] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [13228.519633] LustreError: 414079:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13228.560404] LustreError: 414079:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13232.823447] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13238.760263] LustreError: 414075:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13238.790274] LustreError: 414075:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13241.946744] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13242.427504] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13248.435977] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [13259.232824] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13259.249348] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13259.278652] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13262.816812] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13262.833083] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13262.836419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13265.299783] Lustre: server umount lustre-OST0000 complete [13270.871708] Lustre: server umount lustre-MDT0000 complete [13273.056042] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13273.059047] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13273.108542] LustreError: 414080:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13273.126377] LustreError: 414080:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13276.201965] LustreError: 414059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843802 with bad export cookie 11258496365181216793 [13276.208423] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13276.213225] LustreError: 414059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13276.669187] Lustre: server umount lustre-MDT0001 complete [13291.297413] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [13295.356394] Key type lgssc unregistered [13295.665871] LNet: 417236:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13295.671973] LNetError: 417236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13295.697183] LNet: Removed LNI 192.168.204.120@tcp [13296.829679] Key type .llcrypt unregistered [13296.833565] Key type ._llcrypt unregistered [13329.718635] Key type ._llcrypt registered [13329.720928] Key type .llcrypt registered [13329.820512] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [13331.246862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13331.434822] alg: No test for adler32 (adler32-zlib) [13332.669874] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [13333.031190] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [13334.751599] Key type lgssc registered [13336.298104] Lustre: Echo OBD driver; http://www.lustre.org/ [13346.263237] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13346.287553] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13361.888382] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13367.017686] LustreError: 418626:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.120@tcp: failed processing log, type 4: rc = -110 [13393.759430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13402.364322] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [13412.949185] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13422.998744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13423.690173] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13423.778686] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13428.706378] LustreError: 418651:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13428.761649] LustreError: 418651:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13429.362730] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [13429.762606] LustreError: 419665:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13429.788406] LustreError: 419665:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13434.856388] LustreError: 418652:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13434.891369] LustreError: 418652:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13439.974769] LustreError: 418651:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13439.996777] LustreError: 418651:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13440.887258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13441.451265] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13447.760387] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [13458.590734] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13465.851940] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13474.784774] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13474.806346] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13474.837192] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13477.344771] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13477.345363] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13477.356374] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13477.408239] Lustre: Skipped 1 previous similar message [13480.243452] Lustre: server umount lustre-OST0000 complete [13487.590570] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13487.591844] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13487.663369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13487.677727] Lustre: Skipped 1 previous similar message [13492.707436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13492.717553] Lustre: Skipped 1 previous similar message [13497.825326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13500.383790] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13500.621059] Lustre: server umount lustre-MDT0000 complete [13502.945102] LustreError: 419669:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13502.964773] LustreError: 419669:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13505.337227] LustreError: 418632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844031 with bad export cookie 16048453485455650196 [13505.347381] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13505.358988] LustreError: 418632:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13505.855396] Lustre: server umount lustre-MDT0001 complete [13518.302466] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [13521.529286] Key type lgssc unregistered [13521.955532] LNet: 422203:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13521.973724] LNetError: 422203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13522.011546] LNet: Removed LNI 192.168.204.120@tcp [13523.175071] Key type .llcrypt unregistered [13523.177449] Key type ._llcrypt unregistered [13551.024375] Key type ._llcrypt registered [13551.027740] Key type .llcrypt registered [13551.197807] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [13552.725808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13552.939282] alg: No test for adler32 (adler32-zlib) [13554.023849] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [13554.304492] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [13556.039236] Key type lgssc registered [13557.575584] Lustre: Echo OBD driver; http://www.lustre.org/ [13566.941471] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13566.955532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13568.520911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13574.121771] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [13578.725448] LustreError: 423624:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13578.767859] LustreError: 423624:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13583.846992] LustreError: 423625:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13584.290170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13584.752454] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13589.733662] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [13597.536289] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13603.753484] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13611.956013] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13612.326845] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13613.381615] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13619.531898] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [13627.817876] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13635.052764] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13635.066755] Lustre: Skipped 1 previous similar message [13635.070773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13635.083763] Lustre: Skipped 2 previous similar messages [13639.308280] Lustre: server umount lustre-MDT0000 complete [13640.164687] LustreError: 423621:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13640.190655] LustreError: 423621:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13641.475673] LustreError: 423606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.20@tcp arrived at 1787844167 with bad export cookie 3817801811428069246 [13641.497459] LustreError: 423606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13643.372249] LustreError: 425469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844169 with bad export cookie 3817801811428068357 [13643.377301] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13643.383467] LustreError: 425469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13645.283573] LustreError: 424352:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13645.283825] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13645.284962] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13645.299807] LustreError: 424352:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13645.313029] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13645.349311] Lustre: Skipped 1 previous similar message [13648.291272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13649.575321] Lustre: server umount lustre-OST0000 complete [13653.419156] LustreError: 424352:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13653.433771] LustreError: 424352:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13657.363661] Lustre: server umount lustre-MDT0001 complete [13669.039830] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [13671.790520] Key type lgssc unregistered [13672.078381] LNet: 427312:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13672.088554] LNetError: 427312:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13672.113387] LNet: Removed LNI 192.168.204.120@tcp [13672.875191] Key type .llcrypt unregistered [13672.880890] Key type ._llcrypt unregistered [13697.710625] Key type ._llcrypt registered [13697.715385] Key type .llcrypt registered [13697.830117] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [13699.073284] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13699.174754] alg: No test for adler32 (adler32-zlib) [13700.269591] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [13700.475143] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [13702.177478] Key type lgssc registered [13703.176785] Lustre: Echo OBD driver; http://www.lustre.org/ [13712.340395] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13712.363128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13714.001777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13719.320486] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [13723.128215] LustreError: 428728:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13723.144787] LustreError: 428728:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13728.226876] LustreError: 428729:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13728.614758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13728.873567] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13733.462679] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [13741.209198] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13746.730159] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13752.947827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13753.280317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13758.394368] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13759.991110] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [13768.009170] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13773.804313] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13773.824872] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13773.835951] Lustre: Skipped 1 previous similar message [13775.243232] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.20@tcp (stopping) [13778.914577] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13778.931767] Lustre: Skipped 1 previous similar message [13779.125439] Lustre: server umount lustre-MDT0001 complete [13780.333424] LustreError: 431309:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.20@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13784.034199] LustreError: 428724:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13784.050181] LustreError: 428724:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13789.154019] LustreError: 428729:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13789.157689] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13789.162951] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13789.162962] Lustre: Skipped 1 previous similar message [13789.165694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13793.397509] Lustre: server umount lustre-OST0000 complete [13797.696558] Lustre: server umount lustre-MDT0000 complete [13812.249460] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [13815.488431] Key type lgssc unregistered [13815.826489] LNet: 432418:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13815.833605] LNetError: 432418:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13815.871457] LNet: Removed LNI 192.168.204.120@tcp [13816.842483] Key type .llcrypt unregistered [13816.845144] Key type ._llcrypt unregistered [13846.632822] Key type ._llcrypt registered [13846.640504] Key type .llcrypt registered [13846.860132] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [13863.999841] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [13865.039487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13865.065216] alg: No test for adler32 (adler32-zlib) [13866.071786] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [13866.361269] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [13868.063263] Key type lgssc registered [13869.268142] Lustre: Echo OBD driver; http://www.lustre.org/ [13918.595207] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [13936.237480] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13936.301591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13937.579776] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13937.601188] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13937.672326] Lustre: lustre-MDT0000: new disk, initializing [13937.760371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13937.779833] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13942.978285] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [13961.812992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13962.175411] Lustre: lustre-OST0000: new disk, initializing [13962.180290] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13962.194596] Lustre: Skipped 1 previous similar message [13962.201346] Lustre: 437458:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13962.290829] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13969.685957] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [13970.107950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13970.118101] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13970.157089] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13981.506725] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13998.528226] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13998.781533] Lustre: 436440:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13998.922677] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13999.015238] Lustre: lustre-MDT0001: new disk, initializing [13999.165239] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13999.208482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13999.234628] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [14004.554971] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [14018.347896] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [14018.870317] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [14024.144289] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [14024.539800] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [14032.353898] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14032.357342] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [14032.382921] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14032.420415] Lustre: Skipped 1 previous similar message [14035.362436] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.20@tcp (stopping) [14037.544986] Lustre: server umount lustre-MDT0001 complete [14042.599860] LustreError: 436447:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14042.623881] LustreError: 436447:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [14047.716557] LustreError: 436451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14047.737406] LustreError: 436451:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14052.848606] LustreError: 436446:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14057.440737] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14057.458060] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14057.477075] Lustre: Skipped 1 previous similar message [14057.493653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14057.955579] LustreError: 436451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14057.977368] LustreError: 436451:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14061.264886] Lustre: server umount lustre-OST0000 complete [14063.082230] LustreError: 436450:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14072.084646] Lustre: server umount lustre-MDT0000 complete [14087.142302] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [14090.520755] Key type lgssc unregistered [14090.910149] LNet: 441209:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14090.928765] LNetError: 441209:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14090.946671] LNet: Removed LNI 192.168.204.120@tcp [14092.209929] Key type .llcrypt unregistered [14092.213770] Key type ._llcrypt unregistered [14141.911433] Key type ._llcrypt registered [14141.915787] Key type .llcrypt registered [14142.052828] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [14142.368773] Key type .llcrypt unregistered [14142.372603] Key type ._llcrypt unregistered [14161.723392] Key type ._llcrypt registered [14161.725749] Key type .llcrypt registered [14161.849903] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [14163.280320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14163.313263] alg: No test for adler32 (adler32-zlib) [14164.415631] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [14164.725502] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [14166.407242] Key type lgssc registered [14167.720741] Lustre: Echo OBD driver; http://www.lustre.org/ [14200.314248] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14200.353365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14201.755139] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14201.801730] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14201.868289] Lustre: lustre-MDT0000: new disk, initializing [14201.996620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14202.033033] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14207.744692] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [14219.574392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14219.752750] Lustre: 444593:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14219.818729] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14219.821591] Lustre: Skipped 1 previous similar message [14219.877987] Lustre: lustre-MDT0001: new disk, initializing [14220.007071] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14220.069890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14220.085990] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14224.510657] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [14234.489378] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14241.248830] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14248.551924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14248.966236] Lustre: lustre-OST0000: new disk, initializing [14248.975445] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14249.008721] Lustre: 446564:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14249.145097] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14255.459042] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [14258.246789] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14258.262262] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14258.395883] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14268.177311] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14279.141505] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14279.149264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14279.155474] Lustre: Skipped 1 previous similar message [14279.177263] Lustre: Skipped 2 previous similar messages [14284.261113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14284.284436] Lustre: Skipped 2 previous similar messages [14285.244151] Lustre: server umount lustre-MDT0000 complete [14289.214148] LustreError: 444587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844815 with bad export cookie 9776611723966807359 [14289.242661] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14289.245746] LustreError: 444587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14289.377716] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14289.415212] Lustre: Skipped 1 previous similar message [14289.433772] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14289.751496] Lustre: server umount lustre-MDT0001 complete [14296.350449] Lustre: server umount lustre-OST0000 complete [14307.995264] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 11:33:52 (1787844832) [14317.413317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14317.951116] LustreError: 448592:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14317.974264] LustreError: 448592:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14318.027073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14323.169862] LustreError: 448592:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14323.194718] LustreError: 448592:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14323.290700] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [14327.275351] LustreError: 448593:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14332.384797] LustreError: 448592:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14334.037610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14334.494651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14339.832920] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [14349.041558] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14355.866557] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14375.813275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14376.210943] Lustre: Found index 0 for lustre-OST0000, updating log [14376.546082] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14386.097979] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [14388.891706] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14397.117664] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14404.579870] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14404.595568] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14404.600079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14404.617544] Lustre: Skipped 1 previous similar message [14404.636730] Lustre: Skipped 1 previous similar message [14409.503901] Lustre: server umount lustre-OST0000 complete [14426.601134] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14426.617627] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14426.640199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14426.645587] Lustre: Skipped 2 previous similar messages [14431.929227] Lustre: server umount lustre-MDT0000 complete [14432.226801] LustreError: 448592:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14432.247904] LustreError: 448592:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14436.003644] LustreError: 448574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844961 with bad export cookie 9776611723966808745 [14436.008425] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14436.013306] LustreError: 448574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14436.452779] Lustre: server umount lustre-MDT0001 complete [14440.287101] LNet: 1 local NIs in recovery (showing 1): 192.168.204.120@tcp [14446.822280] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 11:36:10 (1787844970) [15070.494656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15070.854624] LustreError: 455006:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15070.950881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15076.323554] LustreError: 455007:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15076.484195] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [15080.426859] LustreError: 455006:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15086.397981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15086.563808] LustreError: 455007:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15086.596627] LustreError: 455007:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15086.899639] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15092.047593] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [15100.316598] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15106.734516] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15126.578381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15126.907744] Lustre: Found index 0 for lustre-OST0000, updating log [15127.352474] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15131.487104] LNet: 1 local NIs in recovery (showing 1): 192.168.204.120@tcp [15134.582401] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [15144.365407] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15146.823431] Lustre: server umount lustre-OST0000 complete [15152.356193] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15152.358476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15152.384337] Lustre: Skipped 2 previous similar messages [15152.414111] Lustre: Skipped 5 previous similar messages [15153.455917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15153.471371] Lustre: Skipped 1 previous similar message [15155.490983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15155.511355] Lustre: Skipped 3 previous similar messages [15157.539679] LustreError: 455996:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15157.550050] LustreError: 455996:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15157.629438] Lustre: server umount lustre-MDT0000 complete [15162.211747] LustreError: 454987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845688 with bad export cookie 9776611723966809837 [15162.219763] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15162.226357] LustreError: 454987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15162.710664] Lustre: server umount lustre-MDT0001 complete [15191.905356] LNet: 1 local NIs in recovery (showing 1): 192.168.204.120@tcp [15196.601741] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [15207.926502] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [15250.159404] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 11:49:34 (1787845774) [15272.318517] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [15276.023518] Key type lgssc unregistered [15276.425624] LNet: 461628:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15276.433627] LNetError: 461628:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15390.121495] LNet: Removed LNI 192.168.204.120@tcp [15391.358371] Key type .llcrypt unregistered [15391.365212] Key type ._llcrypt unregistered [15408.833453] Key type ._llcrypt registered [15408.842688] Key type .llcrypt registered [15408.968245] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [15410.251530] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15410.438834] alg: No test for adler32 (adler32-zlib) [15411.730919] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [15412.106534] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [15413.839258] Key type lgssc registered [15415.156834] Lustre: Echo OBD driver; http://www.lustre.org/ [15420.340593] LNet: Added LNI 192.168.204.120@tcp100 [8/256/0/180] [15421.221345] LNet: Added LNI 192.168.204.120@tcp101 [8/256/0/180] [15422.288096] LNet: Added LNI 192.168.204.120@tcp102 [8/256/0/180] [15423.183613] LNet: Added LNI 192.168.204.120@tcp103 [8/256/0/180] [15424.226439] LNet: Added LNI 192.168.204.120@tcp104 [8/256/0/180] [15425.362503] LNet: Added LNI 192.168.204.120@tcp105 [8/256/0/180] [15426.418373] LNet: Added LNI 192.168.204.120@tcp106 [8/256/0/180] [15427.513631] LNet: Added LNI 192.168.204.120@tcp107 [8/256/0/180] [15428.603811] LNet: Added LNI 192.168.204.120@tcp108 [8/256/0/180] [15429.936910] LNet: Added LNI 192.168.204.120@tcp109 [8/256/0/180] [15431.246791] LNet: Added LNI 192.168.204.120@tcp110 [8/256/0/180] [15432.380744] LNet: Added LNI 192.168.204.120@tcp111 [8/256/0/180] [15433.589967] LNet: Added LNI 192.168.204.120@tcp112 [8/256/0/180] [15434.885162] LNet: Added LNI 192.168.204.120@tcp113 [8/256/0/180] [15436.120708] LNet: Added LNI 192.168.204.120@tcp114 [8/256/0/180] [15437.188966] LNet: Added LNI 192.168.204.120@tcp115 [8/256/0/180] [15438.601244] LNet: Added LNI 192.168.204.120@tcp116 [8/256/0/180] [15439.856831] LNet: Added LNI 192.168.204.120@tcp117 [8/256/0/180] [15441.112240] LNet: Added LNI 192.168.204.120@tcp118 [8/256/0/180] [15442.161735] LNet: Added LNI 192.168.204.120@tcp119 [8/256/0/180] [15443.277922] LNet: Added LNI 192.168.204.120@tcp120 [8/256/0/180] [15444.437034] LNet: Added LNI 192.168.204.120@tcp121 [8/256/0/180] [15445.715412] LNet: Added LNI 192.168.204.120@tcp122 [8/256/0/180] [15446.690410] LNet: Added LNI 192.168.204.120@tcp123 [8/256/0/180] [15447.761845] LNet: Added LNI 192.168.204.120@tcp124 [8/256/0/180] [15449.077610] LNet: Added LNI 192.168.204.120@tcp125 [8/256/0/180] [15450.370170] LNet: Added LNI 192.168.204.120@tcp126 [8/256/0/180] [15451.370763] LNet: Added LNI 192.168.204.120@tcp127 [8/256/0/180] [15452.255132] LNet: Added LNI 192.168.204.120@tcp128 [8/256/0/180] [15453.188790] LNet: Added LNI 192.168.204.120@tcp129 [8/256/0/180] [15454.104303] LNet: Added LNI 192.168.204.120@tcp130 [8/256/0/180] [15455.125838] LNet: Added LNI 192.168.204.120@tcp131 [8/256/0/180] [15456.237963] LNet: Added LNI 192.168.204.120@tcp132 [8/256/0/180] [15457.381382] LNet: Added LNI 192.168.204.120@tcp133 [8/256/0/180] [15458.463137] LNet: Added LNI 192.168.204.120@tcp134 [8/256/0/180] [15459.673094] LNet: Added LNI 192.168.204.120@tcp135 [8/256/0/180] [15468.767644] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15468.831270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15470.001825] Lustre: 464665:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.120@tcp: can't add NID 192.168.204.120@tcp130: rc = -75 [15470.085347] LNetError: 464686:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.120 [15470.103662] LNetError: Protocol error connecting to 192.168.204.120@tcp117 at host 192.168.204.120:988: is it running a compatible version of Lustre? [15470.115871] LNetError: Skipped 4 previous similar messages [15470.261381] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15470.320810] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15470.437714] Lustre: lustre-MDT0000: new disk, initializing [15470.535717] Lustre: 464723:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.120@tcp: can't add NID 192.168.204.120@tcp130: rc = -75 [15470.548484] Lustre: 464723:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15470.559204] LNet: 5 local NIs in recovery (showing 5): 192.168.204.120@tcp120, 192.168.204.120@tcp121, 192.168.204.120@tcp123, 192.168.204.120@tcp126, 192.168.204.120@tcp127 [15470.612987] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15470.646572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15471.655563] LNetError: 464697:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.120 [15471.676020] LNetError: 464697:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 13 previous similar messages [15471.686546] LNetError: Protocol error connecting to 192.168.204.120@tcp120 at host 192.168.204.120:988: is it running a compatible version of Lustre? [15471.700547] LNetError: Skipped 11 previous similar messages [15473.698320] LNetError: 462123:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.120 [15473.716018] LNetError: 462123:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 3 previous similar messages [15473.722180] LNetError: Protocol error connecting to 192.168.204.120@tcp120 at host 192.168.204.120:988: is it running a compatible version of Lustre? [15473.741785] LNetError: Skipped 1 previous similar message [15476.633218] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [15477.795634] LNetError: 464688:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.120 [15477.816658] LNetError: Protocol error connecting to 192.168.204.120@tcp120 at host 192.168.204.120:988: is it running a compatible version of Lustre? [15485.990371] LNetError: 464687:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.120 [15486.000755] LNetError: Protocol error connecting to 192.168.204.120@tcp120 at host 192.168.204.120:988: is it running a compatible version of Lustre? [15490.318472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15490.479561] Lustre: 464682:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15490.584457] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15490.591871] Lustre: Skipped 1 previous similar message [15490.694954] Lustre: lustre-MDT0001: new disk, initializing [15490.768588] Lustre: 465486:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.120@tcp: can't add NID 192.168.204.120@tcp130: rc = -75 [15490.784202] Lustre: 465486:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15490.927085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15490.975603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15490.991434] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15496.339607] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [15498.313402] Lustre: 465828:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.120@tcp: can't add NID 192.168.204.120@tcp130: rc = -75 [15498.331903] Lustre: 465828:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15500.192909] LNetError: 462121:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.120 [15500.201510] LNetError: Protocol error connecting to 192.168.204.120@tcp125 at host 192.168.204.120:988: is it running a compatible version of Lustre? [15508.362988] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15515.411812] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15516.579919] LNetError: 464691:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.120 [15516.597201] LNetError: 464691:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 14 previous similar messages [15516.616405] LNetError: Protocol error connecting to 192.168.204.120@tcp128 at host 192.168.204.120:988: is it running a compatible version of Lustre? [15516.641830] LNetError: Skipped 14 previous similar messages [15524.477423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15524.840854] Lustre: lustre-OST0000: new disk, initializing [15524.845083] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15524.851628] Lustre: 466659:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15524.958534] Lustre: 466664:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.120@tcp: can't add NID 192.168.204.120@tcp130: rc = -75 [15524.966197] Lustre: 466664:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15525.019859] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15530.031390] LNet: 1 local NIs in recovery (showing 1): 192.168.204.120@tcp120 [15531.640367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15531.672019] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15531.851670] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15533.598441] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [15546.475261] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15549.354706] LNetError: 464684:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.120 [15549.364273] LNetError: 464684:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 29 previous similar messages [15549.370982] LNetError: Protocol error connecting to 192.168.204.120@tcp122 at host 192.168.204.120:988: is it running a compatible version of Lustre? [15549.380095] LNetError: Skipped 29 previous similar messages [15557.608865] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15557.617296] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15557.617671] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15557.637043] Lustre: Skipped 1 previous similar message [15562.661277] Lustre: server umount lustre-OST0000 complete [15572.969922] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15572.985103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15573.022249] Lustre: Skipped 1 previous similar message [15573.031453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15574.505033] Lustre: server umount lustre-MDT0000 complete [15579.391900] LustreError: 466658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846105 with bad export cookie 17031147657720350150 [15579.398768] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15579.412183] LustreError: 466658:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15580.234588] Lustre: server umount lustre-MDT0001 complete [15592.932244] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [15597.117875] Key type lgssc unregistered [15597.585448] LNet: 468577:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15597.593296] LNetError: 468577:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15597.606434] LNet: Removed LNI 192.168.204.120@tcp [15598.623360] LNet: Removed LNI 192.168.204.120@tcp100 [15598.628308] LNetError: 468577:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15598.651526] LNetError: 468577:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15598.669146] LNet: Removed LNI 192.168.204.120@tcp101 [15599.711196] LNet: Removed LNI 192.168.204.120@tcp102 [15599.723985] LNetError: 468577:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15599.733379] LNetError: 468577:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15599.747630] LNet: Removed LNI 192.168.204.120@tcp103 [15600.799277] LNet: Removed LNI 192.168.204.120@tcp104 [15601.823315] LNet: Removed LNI 192.168.204.120@tcp105 [15601.825705] LNetError: 468577:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15601.847266] LNetError: 468577:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15601.871297] LNet: Removed LNI 192.168.204.120@tcp106 [15602.911893] LNet: Removed LNI 192.168.204.120@tcp107 [15603.939256] LNet: Removed LNI 192.168.204.120@tcp108 [15604.959166] LNet: Removed LNI 192.168.204.120@tcp109 [15605.983146] LNet: Removed LNI 192.168.204.120@tcp110 [15605.985262] LNetError: 468577:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15606.004325] LNetError: 468577:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15606.019466] LNet: Removed LNI 192.168.204.120@tcp111 [15607.072174] LNet: Removed LNI 192.168.204.120@tcp112 [15608.095624] LNet: Removed LNI 192.168.204.120@tcp113 [15609.119247] LNet: Removed LNI 192.168.204.120@tcp114 [15610.150593] LNet: Removed LNI 192.168.204.120@tcp115 [15611.168342] LNet: Removed LNI 192.168.204.120@tcp116 [15612.191345] LNet: Removed LNI 192.168.204.120@tcp117 [15613.215252] LNet: Removed LNI 192.168.204.120@tcp118 [15614.239153] LNet: Removed LNI 192.168.204.120@tcp119 [15614.241441] LNetError: 468577:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15614.256629] LNetError: 468577:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15614.263367] LNet: Removed LNI 192.168.204.120@tcp120 [15615.327304] LNet: Removed LNI 192.168.204.120@tcp121 [15616.351881] LNet: Removed LNI 192.168.204.120@tcp122 [15617.375181] LNet: Removed LNI 192.168.204.120@tcp123 [15618.402295] LNet: Removed LNI 192.168.204.120@tcp124 [15619.423681] LNet: Removed LNI 192.168.204.120@tcp125 [15620.447298] LNet: Removed LNI 192.168.204.120@tcp126 [15621.472158] LNet: Removed LNI 192.168.204.120@tcp127 [15622.495595] LNet: Removed LNI 192.168.204.120@tcp128 [15623.519690] LNet: Removed LNI 192.168.204.120@tcp129 [15623.524989] LNet: Removed LNI 192.168.204.120@tcp130 [15623.528057] LNet: Removed LNI 192.168.204.120@tcp131 [15623.531530] LNet: Removed LNI 192.168.204.120@tcp132 [15623.533672] LNet: Removed LNI 192.168.204.120@tcp133 [15623.538784] LNet: Removed LNI 192.168.204.120@tcp134 [15623.563342] LNet: Removed LNI 192.168.204.120@tcp135 [15624.758919] Key type .llcrypt unregistered [15624.761296] Key type ._llcrypt unregistered [15655.826759] Key type ._llcrypt registered [15655.841831] Key type .llcrypt registered [15656.003116] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [15673.092573] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [15674.994288] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15675.241562] alg: No test for adler32 (adler32-zlib) [15676.655298] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [15676.988375] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [15678.767165] Key type lgssc registered [15680.099875] Lustre: Echo OBD driver; http://www.lustre.org/ [15724.283819] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 11:57:28 (1787846248) [15747.220089] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [15750.988660] Key type lgssc unregistered [15751.353815] LNet: 472673:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15751.360650] LNetError: 472673:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15751.381289] LNet: Removed LNI 192.168.204.120@tcp [15752.298553] Key type .llcrypt unregistered [15752.303624] Key type ._llcrypt unregistered [15772.148378] Key type ._llcrypt registered [15772.153982] Key type .llcrypt registered [15772.357788] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [15773.606164] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15773.645786] alg: No test for adler32 (adler32-zlib) [15775.102417] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [15775.379951] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [15777.151545] Key type lgssc registered [15778.809859] Lustre: Echo OBD driver; http://www.lustre.org/ [15790.499433] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15790.533950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15791.835249] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15791.875137] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15791.995523] Lustre: lustre-MDT0000: new disk, initializing [15792.117058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15792.161881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15797.434371] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [15809.861543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15810.096427] Lustre: 473884:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15810.290101] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15810.298683] Lustre: Skipped 1 previous similar message [15810.439587] Lustre: lustre-MDT0001: new disk, initializing [15810.590794] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15810.680794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15810.690790] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15816.786264] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [15828.062915] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15834.851497] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15842.657821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15843.023687] Lustre: lustre-OST0000: new disk, initializing [15843.028618] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15843.051682] Lustre: 475856:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15843.234375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15851.202499] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [15851.678405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15851.694736] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15851.828922] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15862.890555] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15868.226934] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing lnet_if_list [15876.016273] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15876.043739] LNet: Added LNI 192.168.204.120@tcp10 [8/256/0/180] [15884.092854] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15884.129624] LNet: Added LNI 192.168.204.120@tcp11 [8/256/0/180] [15889.900933] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15889.938895] LNet: Added LNI 192.168.204.120@tcp12 [8/256/0/180] [15896.240893] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15896.342561] LNet: Added LNI 192.168.204.120@tcp13 [8/256/0/180] [15903.895226] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15903.941706] LNet: Added LNI 192.168.204.120@tcp14 [8/256/0/180] [15911.427053] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15911.457244] LNet: Added LNI 192.168.204.120@tcp15 [8/256/0/180] [15920.673781] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15920.782954] LNet: Added LNI 192.168.204.120@tcp16 [8/256/0/180] [15928.121123] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15928.162727] LNet: Added LNI 192.168.204.120@tcp17 [8/256/0/180] [15934.992875] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15935.067664] LNet: Added LNI 192.168.204.120@tcp18 [8/256/0/180] [15942.688804] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15942.793204] LNet: Added LNI 192.168.204.120@tcp19 [8/256/0/180] [15948.931773] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15948.967525] LNet: Added LNI 192.168.204.120@tcp20 [8/256/0/180] [15956.830450] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15956.894547] LNet: Added LNI 192.168.204.120@tcp21 [8/256/0/180] [15963.475629] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15963.514868] LNet: Added LNI 192.168.204.120@tcp22 [8/256/0/180] [15970.372442] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15970.403541] LNet: Added LNI 192.168.204.120@tcp23 [8/256/0/180] [15977.073642] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15977.118090] LNet: Added LNI 192.168.204.120@tcp24 [8/256/0/180] [15984.011910] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15984.066775] LNet: Added LNI 192.168.204.120@tcp25 [8/256/0/180] [15990.924281] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15990.965515] LNet: Added LNI 192.168.204.120@tcp26 [8/256/0/180] [15998.634700] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15998.694369] LNet: Added LNI 192.168.204.120@tcp27 [8/256/0/180] [16007.634914] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [16007.704594] LNet: Added LNI 192.168.204.120@tcp28 [8/256/0/180] [16016.835882] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [16016.868952] LNet: Added LNI 192.168.204.120@tcp29 [8/256/0/180] [16025.361084] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [16025.398952] LNet: Added LNI 192.168.204.120@tcp30 [8/256/0/180] [16033.758706] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [16033.796079] LNet: Added LNI 192.168.204.120@tcp31 [8/256/0/180] [16042.560560] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [16042.602898] LNet: Added LNI 192.168.204.120@tcp32 [8/256/0/180] [16051.304485] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [16051.328563] LNet: Added LNI 192.168.204.120@tcp33 [8/256/0/180] [16059.758307] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [16059.790665] LNet: Added LNI 192.168.204.120@tcp34 [8/256/0/180] [16069.252957] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [16069.352106] LNet: Added LNI 192.168.204.120@tcp35 [8/256/0/180] [16078.989201] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [16079.024853] LNet: Added LNI 192.168.204.120@tcp36 [8/256/0/180] [16087.446482] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [16087.482211] LNet: Added LNI 192.168.204.120@tcp37 [8/256/0/180] [16096.234190] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [16096.273987] LNet: Added LNI 192.168.204.120@tcp38 [8/256/0/180] [16105.203495] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [16105.233485] LNet: Added LNI 192.168.204.120@tcp39 [8/256/0/180] [16114.787548] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [16114.867317] LNet: Added LNI 192.168.204.120@tcp40 [8/256/0/180] [16124.760744] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [16124.808705] LNet: Added LNI 192.168.204.120@tcp41 [8/256/0/180] [16133.730169] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [16133.753611] LNet: Added LNI 192.168.204.120@tcp42 [8/256/0/180] [16143.051995] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [16143.081479] LNet: Added LNI 192.168.204.120@tcp43 [8/256/0/180] [16152.942415] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [16152.979720] LNet: Added LNI 192.168.204.120@tcp44 [8/256/0/180] [16164.349117] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [16164.400094] LNet: Added LNI 192.168.204.120@tcp45 [8/256/0/180] [16175.634133] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [16175.670082] LNet: Added LNI 192.168.204.120@tcp46 [8/256/0/180] [16188.119694] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [16188.181456] LNet: Added LNI 192.168.204.120@tcp47 [8/256/0/180] [16204.431213] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [16204.568670] LNet: Added LNI 192.168.204.120@tcp48 [8/256/0/180] [16215.282825] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [16215.350587] LNet: Added LNI 192.168.204.120@tcp49 [8/256/0/180] [16227.764975] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [16227.800483] LNet: Added LNI 192.168.204.120@tcp50 [8/256/0/180] [16249.312905] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16249.320601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16249.328826] Lustre: Skipped 1 previous similar message [16254.433466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16254.447785] Lustre: Skipped 4 previous similar messages [16255.269866] Lustre: server umount lustre-MDT0000 complete [16260.723780] LustreError: 473876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846786 with bad export cookie 10269003770448960482 [16260.744754] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16260.751268] LustreError: 473876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16261.171131] Lustre: server umount lustre-MDT0001 complete [16280.033359] Lustre: 473157:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787846790/real 1787846790] req@00000000d65af2b6 x1874692723113344/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787846806 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16280.083938] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16280.104397] Lustre: Skipped 1 previous similar message [16281.055214] Lustre: 473159:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787846791/real 1787846791] req@000000007e595b38 x1874692723113472/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787846807 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16285.215535] Lustre: 473159:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787846795/real 1787846795] req@00000000c01ca0d0 x1874692723113600/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787846811 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16285.905946] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [16291.296032] Lustre: 473156:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787846801/real 1787846801] req@00000000d9b726ca x1874692723113984/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787846817 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16291.334783] Lustre: 473156:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [16295.880743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16305.983264] LustreError: 475849:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16305.993867] LustreError: 475849:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [16306.039711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16310.064287] LustreError: 475850:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16310.079581] LustreError: 475850:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [16311.205412] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [16314.163869] LustreError: 475849:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16314.187366] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16319.493314] LustreError: 475850:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16321.100631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16321.396373] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16325.507904] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16326.440953] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [16334.593755] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16341.078500] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16361.449660] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16361.454270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16361.469233] Lustre: Skipped 1 previous similar message [16366.578857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16366.600294] Lustre: Skipped 2 previous similar messages [16367.643237] Lustre: server umount lustre-OST0000 complete [16372.704930] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16372.714139] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16372.728849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16376.818890] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16378.312647] Lustre: server umount lustre-MDT0000 complete [16381.923567] LustreError: 500123:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16381.965653] LustreError: 500123:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [16383.299266] LustreError: 473878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846909 with bad export cookie 10269003770448963219 [16383.300348] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16383.320248] LustreError: 473878:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16383.629343] Lustre: server umount lustre-MDT0001 complete [16402.493065] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [16405.339564] Key type lgssc unregistered [16405.782952] LNet: 502954:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16405.795086] LNetError: 502954:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16405.812708] LNet: Removed LNI 192.168.204.120@tcp [16405.814332] LNet: Removed LNI 192.168.204.120@tcp10 [16405.817484] LNet: Removed LNI 192.168.204.120@tcp11 [16405.823571] LNet: Removed LNI 192.168.204.120@tcp12 [16405.830555] LNet: Removed LNI 192.168.204.120@tcp13 [16405.836326] LNet: Removed LNI 192.168.204.120@tcp14 [16405.840597] LNet: Removed LNI 192.168.204.120@tcp15 [16405.846496] LNet: Removed LNI 192.168.204.120@tcp16 [16405.851540] LNet: Removed LNI 192.168.204.120@tcp17 [16405.853873] LNet: Removed LNI 192.168.204.120@tcp18 [16405.858964] LNet: Removed LNI 192.168.204.120@tcp19 [16405.863470] LNet: Removed LNI 192.168.204.120@tcp20 [16405.870379] LNet: Removed LNI 192.168.204.120@tcp21 [16405.873726] LNet: Removed LNI 192.168.204.120@tcp22 [16405.876774] LNet: Removed LNI 192.168.204.120@tcp23 [16405.880296] LNet: Removed LNI 192.168.204.120@tcp24 [16405.883457] LNet: Removed LNI 192.168.204.120@tcp25 [16405.886648] LNet: Removed LNI 192.168.204.120@tcp26 [16405.889659] LNet: Removed LNI 192.168.204.120@tcp27 [16405.892635] LNet: Removed LNI 192.168.204.120@tcp28 [16405.895976] LNet: Removed LNI 192.168.204.120@tcp29 [16405.902909] LNet: Removed LNI 192.168.204.120@tcp30 [16405.906723] LNet: Removed LNI 192.168.204.120@tcp31 [16405.914213] LNet: Removed LNI 192.168.204.120@tcp32 [16405.919303] LNet: Removed LNI 192.168.204.120@tcp33 [16405.923640] LNet: Removed LNI 192.168.204.120@tcp34 [16405.934469] LNet: Removed LNI 192.168.204.120@tcp35 [16405.938596] LNet: Removed LNI 192.168.204.120@tcp36 [16405.941346] LNet: Removed LNI 192.168.204.120@tcp37 [16405.946727] LNet: Removed LNI 192.168.204.120@tcp38 [16405.960374] LNet: Removed LNI 192.168.204.120@tcp39 [16405.966346] LNet: Removed LNI 192.168.204.120@tcp40 [16405.974687] LNet: Removed LNI 192.168.204.120@tcp41 [16405.980488] LNet: Removed LNI 192.168.204.120@tcp42 [16405.985494] LNet: Removed LNI 192.168.204.120@tcp43 [16405.998227] LNet: Removed LNI 192.168.204.120@tcp44 [16406.004093] LNet: Removed LNI 192.168.204.120@tcp45 [16406.010875] LNet: Removed LNI 192.168.204.120@tcp46 [16406.018210] LNet: Removed LNI 192.168.204.120@tcp47 [16406.024202] LNet: Removed LNI 192.168.204.120@tcp48 [16406.029292] LNet: Removed LNI 192.168.204.120@tcp49 [16406.044729] LNet: Removed LNI 192.168.204.120@tcp50 [16406.978377] Key type .llcrypt unregistered [16406.987908] Key type ._llcrypt unregistered [16430.286126] Key type ._llcrypt registered [16430.287513] Key type .llcrypt registered [16430.411916] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [16446.945449] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [16447.684738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16447.919610] alg: No test for adler32 (adler32-zlib) [16449.002772] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [16449.330307] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [16451.048519] Key type lgssc registered [16452.528409] Lustre: Echo OBD driver; http://www.lustre.org/ [16496.258761] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16496.303491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16497.557133] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16497.574793] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16497.629590] Lustre: lustre-MDT0000: new disk, initializing [16497.676660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16497.694174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16502.868469] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [16514.953319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16515.127509] Lustre: 506156:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16515.175826] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16515.180681] Lustre: Skipped 1 previous similar message [16515.270654] Lustre: lustre-MDT0001: new disk, initializing [16515.380597] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16515.427062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16515.440166] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16520.503238] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [16531.694983] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16539.440604] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16549.721887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16549.925354] Lustre: *** cfs_fail_loc=90f, val=0*** [16550.037476] Lustre: lustre-OST0000: new disk, initializing [16550.055704] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16550.068711] Lustre: 508173:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16550.206704] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16557.607786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16557.614803] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16557.718365] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16557.981734] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [16569.086702] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16578.020653] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16578.030077] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16578.038092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16578.066865] Lustre: Skipped 1 previous similar message [16583.146327] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16583.161328] Lustre: Skipped 2 previous similar messages [16588.271412] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16588.282551] Lustre: Skipped 1 previous similar message [16592.351647] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16592.492180] Lustre: server umount lustre-OST0000 complete [16597.474596] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16597.486541] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16597.510607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16598.497801] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16603.346040] Lustre: server umount lustre-MDT0000 complete [16607.669671] LustreError: 506148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847133 with bad export cookie 10473851995035370874 [16607.671559] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16607.679596] LustreError: 506148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16608.181963] Lustre: server umount lustre-MDT0001 complete [16623.225467] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [16628.660595] Key type lgssc unregistered [16629.220701] LNet: 510084:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16629.230316] LNetError: 510084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16629.254852] LNet: Removed LNI 192.168.204.120@tcp [16630.189616] Key type .llcrypt unregistered [16630.192497] Key type ._llcrypt unregistered [16658.836375] Key type ._llcrypt registered [16658.838136] Key type .llcrypt registered [16659.005628] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [16660.413358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16660.514893] alg: No test for adler32 (adler32-zlib) [16661.715211] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [16662.186068] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [16663.976091] Key type lgssc registered [16665.501403] Lustre: Echo OBD driver; http://www.lustre.org/ [16673.701326] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16673.720721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16675.124711] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16679.529257] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [16685.537725] LustreError: 511371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16685.560567] LustreError: 511371:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [16688.331137] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16688.707327] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16692.967363] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [16700.727891] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16706.746836] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16714.946203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16715.501593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16722.765232] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [16731.514659] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16744.134895] loop0: detected capacity change from 0 to 204800000 [16744.322688] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16744.338095] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16744.357040] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16751.483077] loop0: detected capacity change from 0 to 204800000 [16751.690153] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16751.700386] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16751.718898] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16758.686842] loop0: detected capacity change from 0 to 204800000 [16765.414408] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16765.663788] Lustre: fs15246-OST0000: new disk, initializing [16765.668200] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16765.675691] Lustre: 514968:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16765.744974] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16772.642409] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [16783.406367] loop1: detected capacity change from 0 to 204800000 [16790.439383] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16790.667144] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16790.744385] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16790.907486] Lustre: fs15246-MDT0000: new disk, initializing [16791.223957] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16791.275852] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16791.407323] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16792.483879] Lustre: 516092:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000019ed9f0d x1874693653055232/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787847334 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16796.300632] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [16797.551531] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16797.567979] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16797.607835] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16822.755307] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16822.777674] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16825.697723] Lustre: server umount fs15246-MDT0000 complete [16827.884614] LustreError: 511367:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16833.005647] LustreError: 511367:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16835.874772] Lustre: server umount fs15246-OST0000 complete [16851.936718] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16851.950125] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16851.970273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16853.473562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16853.488462] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16858.593358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16858.604972] Lustre: Skipped 2 previous similar messages [16863.199507] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16863.410897] Lustre: server umount lustre-OST0000 complete [16867.808719] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16867.818556] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16867.838769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16873.716530] Lustre: server umount lustre-MDT0000 complete [16873.953614] LustreError: 512100:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16877.767305] LustreError: 511351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847403 with bad export cookie 5078746490287446847 [16877.769328] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16877.788637] LustreError: 511351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16878.280413] Lustre: server umount lustre-MDT0001 complete [16890.014854] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [16893.389123] Key type lgssc unregistered [16893.808986] LNet: 519192:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16893.821242] LNetError: 519192:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16893.852991] LNet: Removed LNI 192.168.204.120@tcp [16895.541951] Key type .llcrypt unregistered [16895.552374] Key type ._llcrypt unregistered [16917.406376] Key type ._llcrypt registered [16917.408274] Key type .llcrypt registered [16917.501190] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [16918.744823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16918.846047] alg: No test for adler32 (adler32-zlib) [16920.020324] Lustre: Lustre: Build Version: 2.17.57_61_g678d580 [16920.385204] LNet: Added LNI 192.168.204.120@tcp [8/256/0/180] [16922.114861] Key type lgssc registered [16924.057770] Lustre: Echo OBD driver; http://www.lustre.org/ [16970.608120] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [16981.352782] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [17024.109540] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [17024.166450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17025.517195] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [17025.555939] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [17025.676796] Lustre: lustre-MDT0000: new disk, initializing [17025.814247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17025.837069] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [17031.607819] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [17043.437193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17043.555761] Lustre: 523663:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [17043.600295] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [17043.605929] Lustre: Skipped 1 previous similar message [17043.668376] Lustre: lustre-MDT0001: new disk, initializing [17043.761264] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17043.816841] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [17043.837233] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [17048.913198] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [17059.120288] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [17066.217846] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [17074.223360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17074.463965] Lustre: lustre-OST0000: new disk, initializing [17074.469596] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [17074.477083] Lustre: 525633:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [17074.565774] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17080.513716] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_default_debug -1 all [17082.417330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [17082.427645] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [17082.515218] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [17089.043956] Lustre: DEBUG MARKER: oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17094.149568] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [17094.315712] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [17099.118907] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [17099.347779] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [17103.331942] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [17103.335766] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17103.342974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17103.346560] LustreError: Skipped 1 previous similar message [17103.369711] Lustre: Skipped 1 previous similar message [17107.300606] Lustre: server umount lustre-OST0000 complete [17113.568799] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17113.581396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17113.584421] Lustre: Skipped 1 previous similar message [17113.600393] Lustre: Skipped 2 previous similar messages [17118.689790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17118.696951] Lustre: Skipped 1 previous similar message [17123.813153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17123.824363] Lustre: Skipped 1 previous similar message [17125.343169] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [17125.518726] Lustre: server umount lustre-MDT0000 complete [17129.221828] LustreError: 523654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847655 with bad export cookie 5068577779901688008 [17129.233396] LustreError: 523654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [17129.234066] LustreError: MGC192.168.204.120@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17129.602786] Lustre: server umount lustre-MDT0001 complete [17154.660696] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing set_hostid [17164.147988] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing load_modules_local [17198.793896] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16871 sec ========= 12:22:02 (1787847722) [17201.108773] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:22:05 (1787847725) === [17205.881476] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:22:09 (1787847729) === [17251.527473] Lustre: DEBUG MARKER: oleg420-server.virtnet: executing unload_modules_local [17254.490637] Key type lgssc unregistered [17254.876888] LNet: 532285:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17254.895538] LNetError: 532285:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17254.931447] LNet: Removed LNI 192.168.204.120@tcp [17255.800464] Key type .llcrypt unregistered [17255.804282] Key type ._llcrypt unregistered